ARI  Contractor  Report  2011-01 


Joint  Measurement  Operations  Controller  (JMOC) 


John  Feeney,  William  Salter, 
Jeanine  Ayers,  Matthew  Puglisi 

Aptima,  Inc. 

Brooke  Schaab 

U.S.  Army  Research  Institute 


This  contractor  report  was  prepared  as  required  by  AR  70-31 ,  under  the  terms  of 
Department  of  the  Army  Contract  number  W91 WAW-07-C-001 1 ,  Aptima,  Inc. 


January  2011 


United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  is  unlimited. 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 


A  Directorate  of  the  Department  of  the  Army 
Deputy  Chief  of  Staff,  G1 


Authorized  and  approved  for  distribution: 


BARBARA  A.  BLACK,  Ph.D  MICHELLE  SAMS,  Ph.D. 

Research  Program  Manager  Director 

Training  and  Leader  Development 


Division 


Research  accomplished  under  contract 
for  the  Department  of  the  Army 

Aptima,  Inc. 

Technical  review  by 

James  W  .Lussier,  U.S.  Army  Research  Institute 


NOTICES 

DISTRIBUTION:  Please  address  correspondence  concerning  distribution  of  reports  to: 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPE-ARI- 
ZXM,  251 1  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926. 

FINAL  DISPOSITION:  This  Contractor  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

NOTE:  The  findings  in  this  Contractor  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT  DATE  (dd-mm-yy) 
January  2011 


4.  TITLE  AND  SUBTITLE 


2.  REPORT  TYPE 

Final 


Joint  Measurement  Operations  Controller  (JMOC) 


6.  AUTHOR(S) 

John  Feeney,  William  Salter,  Jeanine  Ayers,  Matthew  Puglisi 
(Aptima,  Inc.),  and  Brooke  Schaab(U.S.  Army  Research  Institute 
for  the  Social  and  Behavioral  Sciences) 


3.  DATES  COVERED  (from.  .  .  to) 
February  2007  -  September  2009 


5a.  CONTRACT  OR  GRANT  NUMBER 

W91 WAW-07-C-001 1 


5b.  PROGRAM  ELEMENT  NUMBER 

622785 


5c.  PROJECT  NUMBER 

A790 


5d.  TASK  NUMBER 

365 


5e.  WORK  UNIT  NUMBER 


8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 


1343-200 


10.  MONITOR  ACRONYM 

ARI 


11.  MONITOR  REPORT  NUMBER 
Contractor  Report  2011-01 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Aptima,  Inc.,  12  Gill  St,  Ste  1400,  Woburn,  MA  01801 

Army  Research  Institute 

ATTN:  DAPE-ARI-OP,  Bldg  520,  Rm  27 

Aberdeen  Proving  Ground,  MD  21005-5425 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

2511  Jefferson  Davis  Highway 
Arlington,  Virginia  22202-3926 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 
Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

Contracting  Officer’s  Representative  and  Subject  Matter  POC:  Brooke  Schaab 


14.  ABSTRACT  (Maximum  200  words): 

The  increasingly  complex  interactions  of  systems  of  systems  for  training  today’s  military  means  that 
supporting  joint  training  exercises  includes  supporting  live,  virtual  and  constructive  simulations.  Capturing 
and  integrating  observer-based  measures  with  other  data  sources  is  essential  for  supporting  complete 
assessment  of  training  exercises.  The  Joint  After  Action  Review  Repository  Library  (JAAR)  is  a  suite  of 
software  tools,  linked  through  a  common  architecture  that  leverages  a  number  of  efforts  across  Services 
and  is  made  available  to,  and  has  been  used  by,  various  end-user  training  organizations.  Joint 
Measurement  Operations  Control  (JMOC)  successfully  integrated  the  SPOTLITE  platform  for  the 
collection  of  observer-based  measures  into  the  JAAR.  JMOC  also  provided  measures  to  support  both 
exercises  and  the  JAAR  program  by  using  observer-based  measures  to  capture  performance  on  the 
installation  and  training  for  JAAR  integrations  and  deployments.  Finally,  the  efforts  of  JMOC  constitute  a 
foundation  for  transitioning  other  measurement  technologies  into  the  JAAR. 


15.  SUBJECT  TERMS 

JMOC,  SPOTLITE,  JAAR,  measurement  technology 


SECURITY  CLASSIFICATION  OF 

16.  REPORT 
Unclassified 

17.  ABSTRACT 
Unclassified 

18.  THIS  PAGE 
Unclassified 

19.  LIMITATION  OF 

20.  NUMBER 

ABSTRACT 

OF  PAGES 

Unlimited 

46 

21.  RESPONSIBLE  PERSON 
Ellen  Kinzer 

Technical  Publication  Specialist 
703-545-4225 


i 


11 


ARI  Contractor  Report  2011-01 


Joint  Measurement  Operations  Controller  (JMOC) 


John  Feeney,  William  Salter, 
Jeanine  Ayers,  Matthew  Puglisi 

Aptima,  Inc. 

Brooke  Schaab 

U.S.  Army  Research  Institute 


Fort  Leavenworth  Research  Unit 
James  W.  Lussier,  Chief 


This  contractor  report  was  prepared  as  required  by  AR  70-31,  under  the  terms  of 
Department  of  the  Army  Contract  number  W91WAW-07-C-0011,  Aptima,  Inc. 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 


January  2011 


Army  Project  Number  Personnel,  Performance 

622785A790  and  Training  Technology 


Approved  for  public  release;  distribution  is  unlimited. 


Acknowledgement 


The  authors  wish  to  acknowledge  the  contributions  of  many  JFCOM  personnel  who 
supported  this  research  effort.  In  particular,  we  would  like  to  thank  CAPT  Charles  Melcher,  who 
helped  us  get  started  and  gave  initial  direction  to  the  project.  We  would  also  like  to  recognize  the 
Anny  Research  Instituted  (ARI)  personnel  for  their  efforts  to  ensure  the  projecf's  success. 
Finally,  we  express  our  deepest  gratitude  to  Richard  Riggs  and  all  of  the  Joint  After  Action 
Review  Repository  Library  team  members  who  helped  us  to  shape  this  projecf's  success. 


JOINT  MEASUREMENT  OPERATIONS  CONTROLLER  (JMOC) 


EXECUTIVE  SUMMARY 


Research  Requirement: 

Training  for  today"s  military  has  evolved  into  a  complex  interaction  of  systems  of 
systems  including  live,  virtual  and  constructive  simulations.  Therefore,  accurately  recording  the 
range  of  relevant  trainee  behavior  is  becoming  increasingly  challenging.  Capturing  and 
integrating  observer-based  measures  with  system-based  measures  and  data  provides  a  more 
complete  picture  of  trainee  perfonnance.  The  JMOC  project  focused  on  developing  useful 
technology  to  capture  observer  measures  and  comments,  so  key  information  observed  during 
exercises  is  available  for  After  Action  Reviews  and  other  post-exercises  products.  The  purpose 
of  the  current  project  was  to  develop  a  tool  to  enhance  Joint  Exercises  by  facilitating  the 
incorporation  of  observer-based  measures  with  system-based  infonnation  to  enhance  training. 

Procedure: 

Interviews  were  initially  conducted  with  Observer  Trainers  (OTs)  at  Joint  Forces 
Command  (JFCOM)  J7  Operations  Group  to  identify  requirements  for  developing  an  observer 
measures  tool.  Through  the  course  of  these  interviews  a  number  of  issues  were  identified  in 
tenns  of  work  load,  application  to  current  AAR  needs  and  further  research  issues.  While  an 
observer-based  measurement  tool  could  be  useful,  it  would  have  limited  benefits  in  the  current 
operations  environment  due  to  barriers  to  adoption.  As  part  of  this  research  a  demonstration  was 
developed  using  the  SPOTLITE™  tool.  This  demonstration  gained  the  interest  of  the  JFCOM  J7 
Capabilities  Development  Group  which  made  it  possible  to  transition  the  focus  of  JMOC  efforts 
into  integrating  with  the  Joint  After  Action  Review  Repository  Library  (JAAR).  The  JAAR 
provided  both  a  rich  technology  base  and  opportunities  to  work  with  potential  end  users  at  a 
variety  of  training  exercises. 

Findings: 

This  effort  integrated  SPOTLITE  technology  into  the  JAAR,  providing  observer-based 
measurement  capability  to  the  overall  system.  Feedback  from  both  JAAR  engineers  and  end  user 
trainers  was  positive.  They  appreciated  the  ease  of  use  and  felt  the  tool  provided  missing 
observer-based  measurement  capability,  enhancing  their  data  collection  efforts  to  date. 

Utilization  and  Dissemination  of  Findings: 

Through  the  integration  of  SPOTLITE  into  the  JAAR  architecture  the  JMOC  project  now 
makes  observer-based  measures  available  for  incorporation  into  a  number  of  JAAR  components 
and  reporting  functions.  These  observer-based  measurement  capabilities  are  now  available  to  any 
organization  or  component  where  the  JAAR  Suite  is  installed  and  integrated  into  training 
exercises.  In  addition,  the  resulting  SPOTLITE-based  JMOC  technology  can  be  used 
independently  of  the  JAAR. 
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ABSTRACT 


Training  for  today"s  military  has  evolved  into  a  complex  interaction  of  systems  of 
systems  including  live,  virtual,  and  constructive  simulations.  Therefore,  accurately  recording  the 
range  of  relevant  trainee  behavior  is  becoming  increasingly  challenging.  Capturing  and 
integrating  observer-based  measures  with  system-based  measures  and  data  is  necessary  to 
provide  a  complete  picture  of  trainee  performance.  The  JMOC  project  focused  on  developing 
useful  technology  to  capture  observer  measures  and  comments,  so  key  information  observed 
during  exercises  is  available  for  After  Action  Reviews  and  other  post-exercises  products.  JMOC 
successfully  integrated  the  proven  SPOTLITE  platform  for  collecting  observer-based  measures 
into  the  JAAR-RL  (Joint  After  Action  Review  Repository  Library).  JMOC  provided  measures 
for  supporting  military  exercises  and  the  JAAR  program  itself,  by  capturing  perfonnance  on 
installation  and  training  for  JAAR  integrations  and  deployment.  JMOC  accomplishments 
constitute  a  foundation  for  transitioning  other  measurement  technologies  into  the  JAAR.  In 
addition,  the  JMOC  SPOTLITE-based  tool  can  be  used  in  military  training  settings 
independently  of  the  JAAR. 


JOINT  MEASUREMENT  OPERATIONS  CONTROLLER  (JMOC) 


CONTENTS _ 

Page 

Problem . 1 

Approach . 1 

Procedures . 2 

Results . 4 

Discussion . 12 

References . 15 

APPENDIX  A:  JMOC  MAWTS  MEASURES . 16 

APPENDIX  B:  JMOC  -  JAAR  FUNCTIONAL  NEEDS  ASSESSMENT  MAREIX . 17 

APPENDIX  C:  JMOC  -  JAAR  MEASUREMENT  SET . 22 

APPENDIX  D:  AVAILABLE  SPOTLITE  DYNAMIC  TARGETING  CELL  (DTC) 

MEASURES . 24 

APPENDIX  E:  SUMMARY  OF  DIGITAL  PEN/PAPER  TECHNOLOGIES . 38 

LIST  OF  FIGURES 

FIGURE  1:  JMOC  INITIAL  INTEGRATION  DESIGN . 3 

FIGURE  2:  SPOTLITE  MOCK-UP  FOR  JMOC  DEMONSTRATION . 5 

FIGURE  3:  SUMMARY  OF  OBSERVER  BASED  COMMENTS . 6 

FIGURE  4:  PROTOTYPE  ANALYST  TOOLKIT  INTERFACE . 8 

FIGURE  5:  ADDITIONAL  PROTOTYPE  ANALYST  TOOLKIT  INTERFACE  DESIGN . 9 

FIGURE  6:  PROPOSED  INTEGRATION  OF  OBSERVER-BASED  MEASURES  INTO 

JAAR  PLAYBACK . 11 


vii 


viii 


JOINT  MEASUREMENT  OPERATIONS  CONTROLLER  (JMOC) 


Problem 

Currently,  performance  measurement,  assessment,  and  feedback  for  trainees  in  Joint 
training  exercises  rely  heavily  on  observations  made  by  Observer  Trainers  (OTs)  and  Observer 
Controllers  (OCs).  Observers  are  responsible  for  recording  observations  of  trainee  perfonnance 
and  providing  real-time  feedback  during  exercises,  facilitating  After  Action  Reviews  (AARs), 
and  creating  useful  exercise  summary  products.  However,  observer-based  data  collection 
presents  several  challenges.  Using  current  paper  and  pencil  methods,  it  is  difficult  to  collect  and 
organize  observations  efficiently,  to  correlate  and  fuse  observations  in  accordance  with  training 
requirements,  and  to  use  synthesized  data  to  produce  coherent  AAR  and  exercise  products. 
Moreover,  given  the  complexity  and  volume  of  the  data  obtained  by  instrumented  systems  used 
in  training,  the  sheer  number  of  systems  and  observers  involved,  and  the  distributed  nature  off 
live,  virtual,  and  constructed  (LVC)  training  events,  it  is  often  difficult  to  establish  the  exercise 
context  under  which  observations  are  collected.  Thus,  there  is  a  clear  need  for  enhanced 
perfonnance  measurement  and  assessment  capabilities  in  LVC  environments. 

There  is  an  important  pragmatic  aspect  to  the  problem  of  more  effective  and  flexible 
methods  for  collecting  observer-based  data:  OTs  and  OCs  operate  within  an  organizational  and 
operational  context,  with  defined  workflows  and  within  significant  time  and  resource  constraints. 
New  technology  must  be  compatible  with  those  constraints  and  with  the  operating  procedures 
and  expectations  of  the  OTs  and  OCs.  Therefore,  the  problem  the  Joint  Measurement 
Operations  Controller  (JMOC)  project  sought  to  address  combines  technical  and 
operational  aspects.  This  is  not  unique  to  JMOC,  of  course:  successful  development  and 
introduction  of  new  technology  requires  both  the  technology  itself  and  its  adoption,  and  therefore 
the  development  of  the  technology  must  be  informed  by  considerations  of  its  operational 
settings.  It  is  also  important  to  note  that  those  operational  settings  have  important  technical 
components  as  well,  in  terms  of  the  technologies  currently  employed,  and  effective  technology 
development  and  introduction  must  be  responsive  to  those  requirements  as  well. 

Approach 

The  Joint  Measurement  Operations  Controller  (JMOC)  project  required  identification  of  a 
suitable  user  group  so  development  could  focus  on  actual  user  requirements.  It  was  necessary 
that  the  identified  user  group  be  interested  in  potentially  using  JMOC,  since  it  was  neither 
feasible  nor  appropriate  to  attempt  to  impose  a  solution.  JMOC  began  by  focusing  on  computer- 
based  tools  to  facilitate  data  collection  and  analysis  by  OCs  and  OTs  for  Joint  Training  exercises. 
Initial  efforts  were  geared  toward  developing  a  tool  for  use  by  the  OTs  in  the  JFCOM  J7 
Operations  Group  within  their  current  work  environment.  As  work  continued,  the  focus  of  the 
JMOC  project  shifted  to  the  Joint  After-Action  Review  Repository  Library  (JAAR)  activity 
coordinated  by  JFCOM. 

The  JAAR  is  an  engineering  activity  concerned  with  integrating  a  variety  of  existing  and 
new  capabilities  into  a  suite  of  tools  that  can  support  a  variety  of  military  training  exercises. 
JMOC  activities  with  the  JAAR  involved  both  technology  development  and  integration,  on  the 
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one  hand,  and  actual  support  of  field  exercises,  on  the  other,  providing  suitable  exposure  to  end 
user  requirements  as  well  as  a  realistic  context  for  technology  development. 

The  procedures  that  implemented  these  methods  are  discussed  in  more  detail  below. 

Procedures 

To  better  understand  the  needs  of  end-user,  meetings  were  held  with  members  of  the 
JFCOM  J7  Operations  Group  early  in  Phase  II.  These  meetings  were  used  to  define  requirements 
and  establish  expectations  for  JMOC  given  the  organizational  structure  at  JFCOM,  as  well  as  to 
understand  the  hurdles  and  requirements  that  must  be  addressed  for  the  program  to  be  successful. 
Additionally  these  meetings  revealed  to  the  team  the  current  research  being  conducted  on  the 
mapping  of  training  results  to  Doctrine,  Organization,  Training,  Material,  Leadership  and 
Education,  Personnel  and  Facilities  (DOTMLPF)  issues. 

Meetings  were  held  with  JFCOM  J7"s  Capabilities  Development  Group  to  discuss  better 
understand  their  role  with  J7.  It  was  learned  that  the  Capabilities  Development  Group"s  mission 
is  to  develop  future  capabilities  to  improve  joint  training.  These  meetings  were  to  determine  if 
and  how  current  efforts  could  be  transitioned  into  these  future  capabilities.  As  part  of  this 
research  the  JAAR-RL  program  was  introduced  and  their  Engineering  Group  efforts  to  develop  a 
system  of  systems  to  integrate  current  Joint  After- Action  Review  capabilities  based  on  existing 
technologies  in  use  across  services. 

Research  with  both  groups  showed  two  distinct  perspectives.  The  Operations  Group 
focused  on  the  pragmatic  issue  of  ensuring  that  technology  integrates  into  the  users"  workflow 
and  processes.  Additionally,  the  Operations  Group  focused  on  delivering  material  into  the  final 
end  product  of  training  they  saw  as  most  valuable:  the  Commander's  Summary  Report  (CSR). 

The  Capabilities  Group "s  perspective  was  more  focused  on  engineering  and  technology 
considerations  to  collect  and  coordinate  the  varied  data  sources  to  provide  a  more  complete 
review  of  the  training  exercises.  The  JAAR-RL  program  collected  digital  data  in  various  formats 
and  protocols  into  a  central  repository  but  lacked  a  means  to  collect  and  include  observer-based 
measures.  However,  the  JAAR-RL  progranf's  Functional  Needs  Assessments  (see  Appendix  B) 
specifically  identified  requirements  for  observer-based  measures.  JMOC  was  viewed  as  a 
potential  component  to  fulfill  this  need  for  observer-based  measurement. 

After  detailed  assessment  of  these  two  distinct  perspectives  this  effort  focused  on 
integration  with  the  JAAR-RL  project.  The  decision  was  based  on  three  main  factors:  first, 
Aptima"s  SPOTLITE  application  for  collecting  observer-based  measures  had  been  identified  by 
the  JAAR-RL  for  integration,  which  ensured  that  the  basic  requirement  of  a  perceived  user  need 
was  met.  Second,  the  integration  of  existing  technologies  within  the  JAAR-RL"s  architecture 
showed  a  clear  transition  path  to  deployment  of  the  observer-based  measurement  capabilities  to 
support  joint  training  exercises.  And  third,  active  participation  in  the  JAAR  would  involve 
participation  in  real  training  exercises  where  the  technology  would  be  used  and  could  be 
evaluated  by  potential  end  users. 

With  a  new  end-user  identified,  the  team  attended  its  first  the  JAAR  Engineering  Group 
meeting  held  at  the  Joint  Development  and  Integration  Facility  (JDIF)  in  Orlando,  FL  in  January 
2008.  At  the  meeting  the  SPOTLITE  platform  was  introduced,  along  with  the  larger  A-Measure 
suite  and  a  concept  of  measurement  in  training.  We  were  introduced  to  the  JAAR  RL 
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Architecture,  based  on  the  Voyager  Edge  communications  protocol,  intended  to  enable  each 
component  of  the  JAAR  to  connect  and  exchange  information  within  the  JAAR  system. 
Technical  information  for  the  Recursion  Software  Voyager  Edge  platform  was  provided.  The 
initial  design  of  the  integration  of  JMOC  with  the  JAAR  via  Voyager  Edge  is  shown  in  Figure  1. 


Figure  1.  JMOC  Initial  Integration  Design 


Documentation  was  made  on  how  the  SPOTLITE  tool  could  support  the  JAAR  functional 
requirements.  The  provided  documentation  for  the  JAAR  Functional  Needs  Assessment  (FNA) 
was  reviewed  and  a  matrix  of  what  functions  would  be  supported  or  supplieD  by  the  three  main 
components  of  the  JMOC  project  -  SPOTLITE,  Voyager  Edge  (VE),  and  the  Analyst  Toolkit. 
The  detailed  summary  is  in  Appendix  B. 

A  developed's  license  for  VE  was  obtained  from  the  JDIF  to  continue  with  the  integration 
efforts.  Previously  Aptima  was  using  an  evaluation  license  that  was  sufficient  for  the  initial 
testing;  however  the  software  engineers  felt  using  the  full  developed's  license  should  avoid  any 
integration  issues  when  the  product  was  delivered  testing.  Our  team  developed,  tested  and 
integrated  SPOTLITE  with  the  preferred  method  of  using  a  database  VE  agent. 

Integration  efforts  continued  with  the  Recursion  Software  VE  technology.  We 
transitioned  from  the  web  services  interface  to  using  a  VE  database  agent  communicating  with  a 
Postgres  database.  The  JAAR  Engineering  Group  provided  a  data  collector  manager 
specification  and  we  integrated  with  this  updated  specification. 

The  JAAR  integration  and  architecture  meeting  in  May  2008  at  the  JDIF  in  Orlando,  FL. 
Provided  a  venue  to  present  the  existing  capabilities  of  the  JMOC  extended  SPOTLITE  tool  for 
collecting  observer-based  measures.  Additionally,  information  was  provided  on  how  SPOTLITE 
fits  into  the  larger  A-Measure  suite  and  how  the  COMPASS  process  for  developing  measures 
could  be  used  by  the  JAAR  for  both  observer-  and  system-based  measures. 
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As  a  result  of  the  meeting  team  members  provided  documentation  for:  system  integration, 
SPOTLITE  system  requirements,  and  the  Authority  To  Operate  (ATO)  process  as  requested  by 
the  JAAR.  In  addition,  work  continued  with  the  integration  efforts  of  SPOTLITE  with  the  JAAR 
VE  architecture. 


Results 

The  initial  work  on  the  JMOC  project  consisted  of  research  into  applying  technologies 
for  allowing  Observer  Trainers  (OTs)  to  electronically  capture  exercise  observations.  This  work 
included  evaluation  of  electronic  paper  and  handwriting  recognition  software.  Neither  of  these 
technologies  was  sufficiently  robust  to  provide  a  feasible  solution.  Additionally,  incorporating 
these  technologies  would  have  excessively  increased  workload  for  both  the  OTs  and  analysts. 

Existing,  proven  technology  was  leveraged  (Alexander,  et.  al.  2007)  to  demonstrate  how 
an  observer-based  tool  could  be  configured  and  deployed  to  capture  a  rich  set  of  observer-based 
measures  for  use  in  training  exercises.  JFCOM  provided  a  Collection  Management  Plan  (CMP) 
to  develop  a  set  of  measures  for  use  in  the  SPOTLITE™  tool  developed  originally  for  the  Air 
Force  Research  Lab  (and  now  in  use  in  Air  Force,  Anny,  and  Marine  settings). 

Two  initial  prototypes  were  developed  under  the  JMOC  project:  (1)  an  OT  Tool  based  on 
the  SPOTLITE  platfonn  and  (2)  an  Analyst  Tool  for  reviewing  multiple  OT  observations.  These 
tools  were  based  on  review  of  the  JFCOM-provided  Ardent  Sentry  exercise  documentation 
(Commandef's  Summary  Report  and  Collection  Management  Plan)  and  the  initial  discussions 
with  JFCOM  and  the  DOTMLPF  study  presentation. 

Figure  2  shows  the  SPOTLITE  demonstration  with  the  left  side  displaying  the  navigation 
tree  for  the  items  OTs  should  be  mindful  of  during  the  exercise.  The  larger  right  side  of  the 
screen  contains  the  various  prompts  for  the  OT  observations.  Essentially,  the  tool  instantiates  the 
CMP  directives  to  the  OTs  and  provides  the  guidance  and  reference  for  them  to  enter  their 
observations. 
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Figure  2.  SPOTLITE  Mock-up  for  JMOC  Demonstration 


In  addition  to  the  JMOC  OT  Tool,  an  Analyst  Tool  prototype  tool  was  designed.  This 
would  allow  for  additional  functions  relating  to  „scrubbing"  observations  for  use  in  various  end 
products:  AAR,  CSR  and  DOTMLPF.  Once  all  the  observations  are  collected  in  SPOTLITE  and 
downloaded  to  a  repository  the  analyst  would  use  the  a  tool  depicted  in  Figure  3  to  find,  filter 
and  review  the  observations  made  by  OTs  during  the  exercise. 
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Figure  3.  Summary  of  Observer  Based  Comments 


A  demonstration  of  December  2007  of  these  tools  at  JFCOM  Suffolk,  VA  formally 
introduced  the  team  to  the  Joint  After  Action  Review  Repository  Library  (JAAR-RL)  program. 
The  demonstration  feedback  and  follow-on  discussions  were  quite  encouraging.  At  this  meeting 
JMOC"s  focus  and  direction  was  changed.  Our  end-user  was  changed  from  the  JFCOM  J7 
Operations  Group  to  the  JAAR  project.  Our  new  direction  consisted  of  integrating  our  existing 
technology  (SPOTLITE)  into  the  JAAR  Architecture  to  provide  observer-based  data  collection 
and  measures  for  previously  defined  functional  requirements.  The  remainder  of  our  work  in 
JMOC  focused  on  our  newly  identified  end-user  and  the  functional  requirements  of  the  JAAR. 

With  the  adoption  of  any  technology  effort  gaining  acceptance  from  the  user  is  vital  to 
success.  From  our  work  with  the  OTs  it  was  apparent  their  organizational  culture  was  resistant  to 
adopting  new  tools  and  techniques  for  capturing  observer-based  measures  digitally.  While  the 
requirements  identified  -  the  digital  capture,  storage  and  retrieval  of  observer-based  measures  - 
were  technically  feasible,  the  OT  community"s  inability  to  change  their  current  workflow  and 
processes  placed  constraints  on  JMOC  development  efforts  that  could  not  be  overcome. 

The  two  largest  constraints  of  the  OT  community  were  that  the  process  of  recording 
observations  remains  hand- written  and  that  there  could  no  increase  in  current  workload  of  OTs 
and  their  analysts.  Through  an  evaluation  of  the  available  technologies  for  capturing,  storing  and 
retrieving  observer-based  measures  no  technical  solution  was  identified  that  satisfied  these 
constraints.  For  example,  although  digital  pen/paper  would  allow  an  OT  to  digitally  capture 
observation  notes  but  placed  additional  workload  on  the  analysts,  the  effort  needed  to  process  the 
notes  for  use  in  exercise  products  (e.g.  AAR  &  CSR)  and  research/training  enhancements 
violated  the  workload  constraint.  Given  the  results  of  our  research  into  the  various  technologies 
none  of  the  digital  paper/pen  technologies  no  combination  of  products  provided  a  feasible 
solution  for  JMOC.  (See  Appendix  E  for  a  summary  of  Pen/Paper  technologies). 
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JMOC  formally  joined  the  JAAR  Engineering  Group  in  January  2008  and  proceeded  to 
integrate  SPOTLITE™  with  the  JAAR  Architecture,  specifically  integrating  with  Recursion 
Software"s  Voyager  Edge  platfonn  -  a  distributed  communication  protocol  for  establishing, 
securing,  and  performing  communication  in  a  network  of  heterogeneous  devices.  The  first 
integration  effort  included  both  real-time  (wireless/wired)  and  late  join  (batch  upload) 
capabilities.  Both  versions  were  developed  to  ensure  that  real-time  data  collection  could  be 
supported  in  exercises  where  the  network  and  security  constraints  would  allow  such  access  and 
also  ensure  that  data  could  be  captured  when  such  connectivity  was  not  available. 

Given  the  change  in  direction  based  on  the  requirements  of  the  JAAR,  a  proven 
technology  was  selected  to  demonstrate  how  an  observer-based  tool  could  be  configured  and 
deployed  to  capture  a  rich  set  of  observer-based  measures  for  use  in  training  exercises.  A 
JFCOM-provided  Collection  Management  Plan  (CMP)  was  used  to  develop  a  set  of  measures  for 
use  in  the  SPOTLITE™  tool  developed  originally  for  the  Air  Force  Research  Lab  (and  now  in 
use  in  Air  Force,  Army,  and  Marine  settings).  This  existing  observer-based  collection  tool  ™  is  a 
component  of  a  larger  suite  of  capabilities  to  integrate  all  aspects  of  performance  measurement  in 
distributed  training  exercises:  measure  definition;  collection  of  system-based,  observer-based, 
and  self-report  training  data;  analysis;  and  display.  Additional  components  of  this  existing  suite 
of  tools  can  be  integrated  into  the  JAAR  to  help  address  their  important  requirements  for 
assessing,  maintaining,  and  enhancing  military  readiness  and  training  for  both  current  and  future 
forces.  This  is  consistent  with  the  core  approach  being  pursued  by  the  JAAR,  which  is  to 
integrate  and  thereby  leverage  existing  tools  and  technologies,  and  can  provide  solutions  to  a 
number  of  the  JAAR  functional  needs  addressed  in  the  JAAR  Functional  Needs  Assessment. 

SPOTLITE™  was  successfully  integrated  into  the  JAAR  architecture  and  demonstrated 
at  the  Marine  Aviation  Warfare  and  Tactics  Squadron  exercise  (MAWTS)  09-1  in  October  2008. 
For  supporting  testing  and  evaluation  of  observer-based  measures  at  this  service-level  exercise, 
measures  were  developed  based  on  an  existing  data  collection  process  at  MAWTS.  The  measures 
represented  a  key  subset  of  the  total  observations  made  at  the  exercise  and  demonstrated  the  first 
collection  of  observer-based  measures  in  a  JAAR  supported  exercise.  At  the  exercise  the  JMOC 
team  accomplished: 

•  Installing  JMOC"s  SPOTLITE  software  on  the  MAWTS  supplied  Toughbook. 

•  Collected  field  data  in  parallel  with  MAWTS  personnel. 

•  Configured  the  software  to  accomplish  the  late  join  (batch  upload)  for  the  new  IP 
addresses  on  the  MAWTS  equipment. 

•  Successfully  loaded  observer-based  data  from  the  Toughbook  into  the  JAAR 
database. 

Observer  data  was  collected  in  real  time  and  provided  to  the  JAAR  via  the  late -join 
capability  upon  return  from  the  field.  (See  Appendix  A  for  an  Excel  spreadsheet  showing  the 
data  that  were  collected.)  These  data  requirements  were  developed  with  MAWTS  personnel  on¬ 
site  at  the  exercise  and  therefore  are  not  as  detailed  as  measures  developed  using  the  COMPASS 
process  and  implemented  in  SPOTLITE.  This  was  a  successful  and  well-received  demonstration 
of  collecting  observer  data  in  an  exercise  supported  by  the  JAAR. 

In  addition  to  SPOTLITE  integration,  work  included  development  of  initial  concepts  for 
the  Analyst  Toolkit  interface.  The  Analyst  Toolkit  would  enable  analysts  to  monitor,  search,  and 
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filter  observer-based  measured  collected  in  an  exercise.  Figure  5  is  a  prototype  interface 
developed  and  shown  to  the  JAAR  Engineering  Group. 

While  the  SPOTLITE  tool  is  integrated  with  the  JAAR  Architecture  it  physically  resides 
on  individual  observer  tablet  PCs  or  PDAs.  The  SPOTLITE  user  interface  is  independent  of 
other  applications  residing  within  the  JAAR  Workstation.  In  contrast,  the  Analyst  Toolkit  resides 
on  and  is  operated  at  the  JAAR  Workstation.  Prior  to  the  integration  of  SPOTLITE"s  observer- 
based  measurement  capability  there  was  no  need  for  an  analyst  tool  to  support  those  measures.  A 
number  of  interface  designs  were  produced  to  illustrate  how  an  analyst  tool  could  be  used  to 
find,  filter  and  explore  the  observer-based  measures.  Figures  4  and  5  show  Analyst  Toolkit 
prototype  designs  developed  for  the  JAAR.  While  the  prototypes  for  the  Analyst  Toolkit  were 
well  received  further  development  was  postponed  in  favor  of  other  higher  priority  tasks  for  the 
JAAR  Workstation. 


Figure  4.  Prototype  Analyst  Toolkit  Interface 
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Figure  5.  Additional  Prototype  Analyst  Toolkit  Interface  Design 


The  main  efforts  for  integration  of  SPOTLITE  into  the  JAAR  architecture  included  the 
following  software  development  and  testing  tasks: 

1 .  Refactored  SPOTLITE  to  persist  data  to  a  target  database  environment  based  on  a 
parameterized  set  of  variables. 

2.  Migrated  the  OBSERVER  DB  schema  from  SQL  Server  to  PostgreSQL 

3.  Integrated  SPOTLITE  with  the  JAAR  Data  Collector  Manager  (DCM)  software 

a.  SPOTLITE  can  be  launched  from  the  JAAR  DCM  and  accepts  parameters  passed 
from  the  JAAR 

b.  SPOTLITE  persists  data  to  its  own  schema  within  the  JAAR  database 

c.  SPOTLITE  captures  and  persists  the  event  and  session  data  and  associates  its 
observations  with  the  unique  (event  and  session  ids)  JAAR  identifiers 

4.  Implemented  all  the  abstract  methods  of  the  JAAR  Abstract  Data  Collector  so  that 
SPOTLITE  can  respond  to  the  requests  and  instructions  of  the  JAAR  operators. 

5.  Updated  SPOTLITE  for  the  new  JAAR  Communication  Layer  that  superseded  the 
Voyager  Edge  protocol. 
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Additionally  as  part  of  the  JAAR  Engineering  Group,  numerous  requests  were  addressed, 
including  the  following: 

1 .  Provided  complete  IATO  documentation  for  SPOTLITE  and  its  component  software  to 
the  JAAR  manager. 

2.  Provided  an  Aptima  Panasonic  Toughbook  for  use  at  the  JDIF  while  procurements  for 
JDIF"s  Toughbooks  are  underway. 

3.  Provided  application  installation  instructions  and  supporting  documentation. 

4.  Provided  various  software  engineering  documentation  for  JAAR  programmatics  -  e.g. 
SLOC  measure. 

After  the  success  at  MAWTS,  the  JMOC  project  focused  on  integration  into  the  new 
JAAR  Communication  Layer  (JCL)  being  actively  developed  internally  by  the  JAAR  at  the  Joint 
Development  and  Integration  Facility  (JDIF).  The  JCL  was  designed  to  replace  the  Voyager 
Edge  communication  protocol.  This  integration  effort  consisted  of  a  number  of  iterations  as  the 
JCL  evolved  and  matured.  The  result  of  this  effort  concluded  with  successful  participation  in  the 
integration  event  held  at  the  JDIF  in  May,  2009.  Additionally,  while  at  the  JDIF  Aptima  worked 
with  JDIF  personnel  to  develop  and  implement  a  set  of  measures  to  assess  the  installation  of  the 
JAAR  suite.  This  version  of  SPOTLITE™  allows  the  JAAR  Engineering  Group  to  observe  and 
evaluate  the  JAAR  installations  at  any  site  and  have  the  data  available  for  post-hoc  and  trend 
analysis.  This  will  help  make  the  installation  process  more  effective  and  efficient.  Acceptance  of 
the  SPOTLITE  technology  and  associated  measurement  approach  represents  a  gratifying  level  of 
acceptance  by  the  JAAR  community  of  top-caliber  engineers.  They  have  come  to  believe  that  the 
technology,  and  the  measurement  and  analysis  it  makes  possible,  can  add  value  to  their  own 
efforts,  an  endorsement  quite  distinct  from  those  of  end  users  in  the  training  world. 

Integration  with  the  newest  version  of  the  JAAR  Communication  Layer  (JCL)  was 
completed  by  improving  the  user  experience  via  adjusting  where  certain  communications  with 
the  JAAR  occurs.  In  the  previous  version  of  the  JAAR  communication  layer  using  Voyager 
Edge,  the  communicating  began  after  loading  the  setup  page.  In  the  JCL  version  of  the  conops 
the  interactions  happens  sooner.  Earlier  communication  was  established  in  the  application'^ 
sequence  timeline  to  improve  the  user  experience. 

For  the  majority  of  the  JMOC  project  the  main  focus  of  the  JAAR  was  on  collection  from 
various  data  sources,  (e.g.  HLA,  TENA,  DIS,  SPOTLITE,  etc.).  From  an  engineering 
perspective,  focusing  on  system-based  or  observer-based  measures  was  premature  until  data 
collectors  were  established  for  the  JAAR.  The  JMOC  teanf's  expertise  in  measures  and 
measurement  development,  while  recognized,  was  not  fully  utilized  due  to  the  efforts  in 
establishing  the  data  collectors.  However,  in  January  and  February  2009  the  JMOC  team 
participated  in  a  series  of  JAAR  Use  Case  development  discussions  and  meetings  where  the 
focus  shifted  from  data  collection  to  measurement.  The  team  presented  how  to  incorporate  the 
observer-based  measures  into  the  set  of  JAAR  reporting  tools.  These  ideas  included: 

•  Establishing  a  Java  measure  tree  plugin  to  JAAR  Playback  tool  that  would  allow  for  the 
following  functionality: 

o  Time-stamped  display  of  measures  that  could  drive  playback  from  a  tree  (using 
JCL  playback  messages) 

o  Timeline  tool  where  indicators  of  measures  show  up  on  timeline. 
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o  The  concept  would  apply  to  both  observer-based  and  system-based  measures 
when  they  are  available. 

Figure  6  shows  how  observer-based  measures  could  be  integrated  into  a  JAAR  playback 
capability.  The  analyst  would  be  able  to  select  measures  from  the  top  left  hand  side  of  the 
display.  Overlays  of  various  data  sources  are  selected  on  the  lower  left  of  display.  The  time  line 
and  playback  are  on  the  right  side  of  the  display.  Measure  icons  are  lower  right  below  the 
playback  controls. 
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Figure  6.  Proposed  Integration  of  Observer-Based  Measures  into  JAAR  Playback 


In  addition  to  the  software  development  tasks  performed,  the  JMOC  team  worked  in  three 
different  areas  for  integration  of  observer-based  measures  within  the  JAAR:  Joint  Training 
exercises,  a  service  level  exercise,  and  for  internal  use  by  the  JAAR  Engineering  Group. 

For  supporting  testing  and  evaluation  of  observer-based  measures  JMOC  supported  the 
service  level  exercise  at  the  USMC  Marine  Air  Weapons  and  Tactics  Squadron  (MAWTS). 
Measures  were  developed  based  on  an  existing  data  collection  process  at  MAWTS.  The 
measures  represented  a  key  subset  of  the  total  observations  made  at  the  exercise  and 
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demonstrated  the  first  collection  of  observer-based  measures  in  a  JAAR  supported  exercise.  The 
exercise  accomplished: 

•  Installing  JMOC"s  SPOTLITE  software  on  the  MAWTS  supplied  Toughbook. 

•  Collected  field  data  in  parallel  with  MAWTS  personnel  (See  Appendix  A). 

•  Configured  the  software  to  accomplish  the  late  join  (batch  upload)  for  the  new  IP 
addresses  on  the  MAWTS  equipment. 

•  Successfully  loaded  observer-based  data  from  the  Toughbook  into  the  JAAR 
database. 

No  fonnal  evaluation  of  SPOTLITE  was  performed  with  the  JAAR  Engineering  team.  As 
noted  above,  the  focus  of  the  JMOC  project  was  on  integration  with  and  support  of  the  JAAR. 
Several  JMOC  program  accomplishments  serve  as  explicit,  face-valid  indicators  of  positive 
assessments  by  the  JAAR: 

1)  SPOTLITE  was  integrated  into  the  JAAR; 

2)  SPOTLITE  was  used  to  support  a  number  of  exercises  the  JAAR  was  involved  in; 

3)  discussions  have  been  held  about  increasing  SPOTLITE  use  within  the  JAAR-RL 
program  and  integrating  additional  software  components  associated  with  SPOTLITE; 
and 

4)  SPOTLITE  measures  were  developed  for  use  by  the  JAAR  itself  to  enhance  JAAR 
installation  procedures,  as  discussed  below. 

To  support  the  JAAR  Engineering  Group  a  set  of  measures  were  developed  for  the 
installation  and  integration  of  JAAR  suite.  By  establishing  a  set  of  measures  for  use  in 
SPOTLITE  engineers  would  have  a  stable  set  of  measures  when  delivering  and  training  users  of 
the  JAAR  suite.  This  version  of  SPOTLITE™  allows  the  JAAR  Engineering  Group  to  observe 
and  evaluate  the  JAAR  installations  at  any  site  and  have  the  data  available  for  post-hoc  and  trend 
analysis.  This  will  help  make  the  installation  process  more  effective  and  efficient.  Using 
JMOC/SPOTLITE  for  training  of  the  JAAR  users  could  expose  trends  and  areas  for  improving 
training  as  the  JAAR  is  fielded  and  updated.  Appendix  C  has  the  complete  set  of  measures 
developed  for  the  JAAR. 

The  JMOC  project  accomplished  its  overall  goal  of  enhancing  the  use  of  observer- 
measures  for  joint  exercises.  Establishing  a  reliable  and  effective  solution  to  incorporating 
observer-based  measures  by  integrating  existing  proven  technologies  with  the  Joint  After  Action 
Review  Repository  Library  (Riggs  &  Abbott,  2008)  supports  not  only  joint  training  exercises  but 
any  training  supported  by  a  JAAR  application  suite,  which  will  becoming  used  more  widely 
across  services. 


Discussion 

Given  the  increasingly  complex  interactions  of  systems  of  systems  for  training  today"s 
military,  supporting  joint  training  exercises  includes  supporting  live,  virtual  and  constructive 
simulations.  Capturing  and  integrating  observer-based  measures  with  other  data  sources  is 
essential  for  supporting  complete  assessment  of  training  exercises.  The  JMOC  project 
successfully  integrated  proven  technology  -  the  SPOTLITE  platform  for  the  collection  of 
observer-based  measures  -  into  the  JAAR.  JMOC  also  provided  measures  for  supporting 
exercises  and  to  support  the  JAAR  program  itself  by  using  observer-based  measures  to  capture 
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performance  on  the  installation  and  training  for  JAAR  integrations  and  deployments.  Finally,  the 
efforts  of  JMOC  constitute  a  foundation  for  transitioning  other  measurement  technologies  into 
the  JAAR. 

Results  of  the  JMOC  project  support  several  general  conclusions.  First,  development  of 
useful  tools  for  collection  of  observer-based  measures  (or,  indeed,  for  other  purposes)  must  be 
sensitive  to  a  variety  of  technical  and  non-technical  constraints  beyond  the  basic  requirement  that 
it  address  user  needs.  Naturally,  the  technology  must  be  technically  compatible  with  the  target 
environment.  In  addition,  it  must  be  consistent  with  the  organizational  and  procedural  aspects  of 
the  target  environment  -  the  workflows,  standard  operating  procedures,  experience  and  training 
of  the  target  users,  and  the  like.  The  original  target  group  -  OTs  and  OCs  -  had  well-developed 
paper-based  workflows,  and  was  not  a  suitable  target,  given  the  time  and  resource  constraints 
JMOC  faced.  With  transition  to  the  JAAR,  JMOC  was  able  to  leverage  the  considerable 
organizational  and  engineering  work  that  had  already  been  done  in  the  JAAR.  In  addition,  the 
JAAR"s  fimction  is  to  bring  technology  to  bear  to  support  actual  exercises,  so  the  JAAR 
provided  a  smooth  path  to  using  and  evaluating  JMOC  with  real  end  users. 

Two  kinds  of  “cultural”  obstacles  remain,  however,  for  JMOC  (or  other  efforts)  to  deploy 
the  advances  of  human  perfonnance  measurement  in  an  operational  training  setting.  First,  the 
engineering  community  -  as  exemplified  by  the  engineers  in  the  JDIF  -  focuses  on  engineering 
details,  and  is  not  concerned  with  issues  of  measure  development  and  validation.  For  example, 
whether  JMOC-captured  measures  are  placed  in  the  JAAJ  database  is  an  engineering  issue,  but 
whether  those  measures  get  at  important  aspects  of  human  performance  in  training  is  not.  On  the 
other  hand,  the  operational  training  community  -  those  who  design  and  conduct  exercises  -  will 
tend  to  have  more  appreciation  for  the  measurement  issues  involved  but  often  lack  the  time  and 
resources  to  develop  and  test  measures  to  ensure  that  they  both  address  training  requirements  and 
have  the  necessary  metric  properties.  Since  JMOC  (and  much  of  the  rest  of  the  JAAR)  can 
support  existing  measures,  end  user  measures  can  usually  be  incorporated,  enabling  the 
measurement  technology  to  be  deployed.  But  -  to  put  it  bluntly  -  if  the  wrong  things  were  being 
measured,  the  technology  will  still  measure  the  wrong  things,  and  there  is  a  danger  that  it  will 
even  help  to  perpetuate  those  measures. 

On  the  other  hand,  JMOC  and  related  technologies  facilitate  the  incorporation  of  new 
measures,  and  create  opportunities  for  measurement  enhancements,  since  the  technologies 
themselves  are  not  tied  to  a  particular  set  of  measures.  Indeed,  further  JMOC-related  activities 
might  well  leverage  the  inroads  that  the  JAAR  has  made  with  elements  of  the  operational 
training  community  to  work  with  such  users  on  measure  development.  In  addition,  a  variety  of 
measures  have  been  developed  and  validated  for  military  training  communities  across  the 
services  and  implemented  in  SPOTLITE,  which  can  be  readily  used  by  JMOC  users.  (Appendix 
D  presents  some  of  these.) 

As  LVC  exercises  become  more  complex,  driven  both  by  more  complex  missions  and 
associated  training  requirements,  and  by  the  increasing  complexity  of  the  technologies  used  by 
the  military  for  both  training  and  in  deployment,  the  need  for  richer  measures,  and  better  ways  to 
analyze  the  results  of  measurement  and  provide  feedback  to  learners,  will  only  increase. 
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Appendix  A:  JMOC  MAWTS  Measures 


MAWTS  EXCEL  Fields 

Explanation  of  Values 

DT-WTI  Class  Specific 

Desert  Talon  (DT)  or  Weapons  Tactics 
Instructor  (WTI) 

Date 

Call  Sign 

E.G,  Latch  67 

IP-STUD-Blue-Aug 

Instructor  Pilot  (IP),  Student  (STUD),  Some 
other,  non-MAWTS,  aircraft  (BLUE), 
Augment  Instructor  (Aug) 

Name* 

Pilot  Name 

Event 

E.G,  Offensive  Air  Support  2  (OAS-2) 

FAC  Call  Sign 

or  Air  Officer  (AO) 

FAC  Inst  STUD 

Instructor  (Inst),  Student  (STUD) 

T/M/S* 

Type/Model/Series  of  aircraft 

BUNO#* 

Bureau  number  of  aircraft 

#  CREW 

in  aircraft 

Range 

E.G.,  Yodaville.  Range  at  Yuma  where  the 
event  is  being  held. 

Weapon  Specific 

E.G.,  25mm 

Weapon  General 

E.G.,  guns 

Lasing  source 

E.G.,  Ground  Lase 

Lasing  C-D 

Continuous  (C);  Discontinuous  (D) 

FAC 

E.G.,  FAC-A  (airborne  FAC) 

Coord 

Who  provides  coordination 

Release 

Y/N--was  the  weapon  released? 

POA-POI  (m) 

Point  of  Aim/Point  of  lmpact--how  far  did 
the  weapon  land  from  the  target? 

Function 

AH  Dist  To  TGT 

AH  Miss 

AH  LOAL  LOBL 

AH  Pilot  Missile  Fired  Life 

Primary  Reason  Code 

Sec  Reason  Code 

2  Primary  Reason  code 

2  Sec  Reason  code 

Other  Explanation 

3_cat 

Highlighted  fields:  Maj  Rascon  want  the 

‘Indicates 

JAAR  to  capture  these  values 
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Appendix  B:  JMOC  -  JAAR  Functional  Needs  Assessment  Matrix 


Functional  Area 


6.1.1  System 

Operations 

Function 


Item  SPOTLITE  Voyager  Analyst 
Edge  Toolkit 


Description 


1  X 

2  X  X 

3  X  X 

5  X 

6  X 


The  systems  operation  function  contains  those  needs  which 
pertain  to  the  general  operating  functions  necessary  to  meet  the 
operational  needs  environment  in  which  the  AAR  tool  set  will 
perform  its  functions. 

The  AAR  system  shall  be  able  to  operate  in  both  a  fixed  and  deployable 
configuration. 

The  AAR  system  shall  be  modifiable  and  extensible  enabling  AAR 
products  to  be  changed  or  additional  products  developed  to  meet 
emerging  requirements. 

The  AAR  system  shall  be  an  open  architecture  to  permit  scalability 
change,  extensibility  and  capability  additions  without  extensive  baseline 
change. 

The  AAR  system  shall  be  expandable  and  designed  to  easily  incorporate 
upgrades  of  capabilities,  functions,  and  technology  to  support  future 
growth  in  doctrine,  equipment,  and  training  systems. 

The  AAR  tool  set  shall  provide  modular  functions  and  generic  tools 
configurable  for  specific  applications,  to  allow  flexibility  in  meeting  the 
needs  of  diverse  groups  of  users  and  audiences  in  different  contexts. 


7 

8  X 

9  X 

10  X 

11  X 

12  X 

13  X 

14  X 

15  X 

16  X 

17  X 

18  X 

19 

23 


X 


X 

X 


X 


The  AAR  tool  set  shall  provide  for  security  up  to,  and  including,  the 
SECRET  level.  The  capability  will  provide  for,  and  allow,  multiple  levels 
of  access  releasable  to  coalition  and  allied  forces. 

X  The  AAR  tool  set  shall  be  required  to  operate  in  the  windows  or  windows 
like  operating  suite  with  direct  synchronization  to  Word,  Excel, 
PowerPoint,  Access,  Outlook,  etc. 

X  The  AAR  tool  set  shall  be  required  to  use  a  “flash  card”  type  capability  to 
back  up  and  store  all  the  collected  data  and  have  the  ability  to  be 
declassified  through  a  series  of  simple  procedures. 

X  The  AAR  tool  set  shall  be  required  to  allow  users  to  create  customized 
tables  and  forms  and  to  upload  data  collected  to  a  database  running  on 
a  desktop  PC  or  server  system  and  use  those  databases  to  generate 
reports  of  preliminary  findings  and  integrate  with  other  software  to 
provide  readily  available  reports. 

X  The  AAR  tool  set  shall  be  required  to  provide  for  strategic,  operational 
and  tactical  analysis. 

The  AAR  tool  set  shall  be  required  to  support  multiple  levels  of  fidelity, 
from  the  Combatant  Commander  to  the  individual  entity  in  a  tactical 
situation. 

The  AAR  tool  set  shall  be  required  to  support  all  levels  of  fidelity,  from 
the  task  group  to  the  individual  war  fighter  in  an  urban  operations 
situation. 

X  The  AAR  tool  set  shall  be  required  to  run  on  the  JTEN  and  the  training 
equipments  used  for  the  event. 

The  AAR  tool  set  shall  provide  for  multi-level  security  and  access 
control. 

X  The  AAR  tool  set  shall  provide  feedback  to  a  detailed  fidelity  at  the 
individual,  company  and/or  battalion  level. 

X  The  AAR  tool  set  shall  provide  complete  after  action  review  for  the  total 
training  environment,  including  all  entities,  be  it  live,  virtual  or 
constructive. 

X  The  AAR  tool  set  shall  be  used  subsequent  to  local  AAR  reviews  at  each 
of  the  sites  that  participated  in  the  event. 

The  AAR  tool  set  shall  be  a  distributed  system  providing  data, 
information  and  knowledge  across  the  JTEN  network. 

The  AAR  tool  set  shall  integrate  various  tools  so  that  raw  performance 
data  can  be  automatically  transformed  into  a  finished  AAR  knowledge  or 
information  product. 
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Functional  Area 


Item  SPOTLITE  Voyager  Analyst 
Edge  Toolkit 


Description 


24 

X 

X 

X 

The  AAR  tool  set  shall  perform  user  authentication  for  its  clients. 

25 

X 

X 

The  AAR  tool  set  shall  provide  capabilities  for  developing  and 
distributing  team  measurement  tools,  diagnostic  assessments,  and  after 
action  review. 

26 

X 

X 

The  AAR  tool  set  shall  provide  database  capabilities  for  storage  and  use 
of  historical  performance  data  collected  during  scenario  runs. 

27 

X 

The  AAR  tool  set  shall  have  the  capability  to  collect  data,  information 
and  knowledge  from  service  specific  AAR  tool  sets. 

28 

X 

X 

The  AAR  tool  set  shall  maximize  utilization  of  information  or  knowledge 
derived  from  service  AAR  tool  sets  to  build  the  joint  perspective. 

29 

X 

X 

X 

The  AAR  tool  set  shall  be  utilized  for  joint  training  exercises  including 
experiments  and  demonstrations  of  new  capabilities  in  a  joint 
environment. 

30 

X 

X 

The  AAR  tool  set  is  required  to  provide  the  functionality  to  accomplish 
total  mission  diagnostic  evaluation  supporting  data,  information  and 
knowledge. 

6.1.2  Operator 

The  operator  interface  function  defines  the  needs  for  the  human  to 

Interface  Function 

machine  interface  with  the  AAR  tool  set.  The  general  need  is  to 
provide  data,  information  or  knowledge  to  the  operators  in  near 
real  time,  or  post  event  at  multiple  locations  concurrently.  There  is 
the  need  for  specific  information  at  real  time,  near  real  time  and 
post  mission  for  the  operators  and  the  training  audience  to  work 
autonomously  or  in  collaboration. 

1 

X 

X 

The  operator  interface  function  shall  provide  an  interface  to  multiple 
operators  at  multiple  sites. 

3 

X 

The  operator  interface  function  shall  provide  for  the  ability  to  use 
Personal  Digital  Assistants  (PDAs)  to  input  non-electronic  data  and 
observer  comments. 

4 

X 

X 

The  operator  interface  function  shall  be  capable  of  uploading  JTA 
observer  measures  to  any  PDA  for  use  during  the  event. 

5 

X 

X 

The  operator  interface  function  shall  allow  for  any  observer  to  input  or 
modify/create  measures  on  the  PDA. 

6 

X 

X 

The  operator  interface  function  shall  have  a  Windows  look  and  feel  is 
user-friendly,  providing  drop-down  menus  and  familiar  human-machine 
interfaces. 

8 

X 

X 

The  operator  interface  function  shall  provide  screen  configurations 
tailorable  for  varied  operating  locations,  facilities,  and  audiences. 

26 

X 

The  operator  interface  function  shall  be  able  to  display  tabular  reports 
containing  processed  data  from  the  data  repository. 

28 

X 

X 

The  operator  interface  function  provide  for  automatic  alerts/notifications 
of  scenario  anomalies,  instrumentation  failures,  and  digital 
communication  failures  at  their  occurrence. 

29 

X 

X 

The  operator  interface  function  shall  provide  the  operator  with  the 
necessary  status  and  controls  to  monitor  the  conduct  of  the  event. 

6.1.3  Planning 

The  planning  function  fulfills  the  needs  to  provide  the  analysts  and 

Function 

training  audience  the  opportunity  to  preset  certain  parameters  for 
measurement,  the  configuration  of  the  system  for  the  event  and  the 
real  time  needs  that  are  necessary  as  feedback  to  the  operators  for 
decision  making  on  the  event  as  it  unfolds.  The  definition  of  the 
Joint  Task  Articles  to  be  executed  will  provide  for  the  data  needs 
by  the  AAR  tool  set  to  provide  the  operators  AAR  summary  reports 
at  the  conclusion  of  the  event. 

15 

X 

The  planning  function  shall  provide  all  data  and  information  for 
recording. 

16 

X 

X 

The  planning  function  shall  allow  for  the  retrieval  of  lessons  learned 
information  from  the  JNTC  libraries. 
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Functional  Area 


Item  SPOTLITE  Voyager  Analyst 
Edge  Toolkit 


Description 


6.1.4  Data  Retrieval 
Function 


1 


5 

9 

10 

11 

15 

16 


6.1.5  Data  Fusion 
Function 


1  U 

3  X 

4  X 


6.1.6 

Data/Knowledge 

Distribution 

Function 


1  X 

2  X 

3  X 

4  X 


X 


The  data  retrieval  function  shall  provide  the  necessary  processing 
to  meet  the  needs  of  obtaining  reference  data  base  information 
from  various  sources,  obtain  the  local  Service  AAR  information  or 
knowledge  outputs,  receive  distributed  joint  AAR  data,  information 
or  knowledge,  and  provide  for  the  filtering  of  irrelevant  data. 

The  data  retrieval  function  shall  be  able  to  access  databases  at  a  rate 
sufficient  to  support  all  aspects  of  the  AAR  functions.  Examples  of  the 
databases  which  should  be  accessible  include,  but  are  not  limited  to,  the 
following: 

X  b.  Intelligence  Databases 

X  The  data  retrieval  function  shall  be  able  to  produce  meta-data,  or  metric 
information,  for  all  received  or  fused  files. 

X  The  data  retrieval  function  shall  be  able  to  establish  filters  to  define 
selected  data  for  inclusion  or  exclusion. 

X  The  data  retrieval  function  shall  be  able  to  retrieve  archived  data  based 
on  established  parameters. 

X  The  data  retrieval  function  shall  be  able  to  access  multiple  databases. 

X  The  data  retrieval  function  shall  have  the  capability  to  reference  and 

communicate  with  various  local  Service  AAR  systems  for  specific  data 
retrieval. 

X  The  data  retrieval  function  shall  have  the  capability  to  retrieve  data  from 
the  joint  distributed  AAR  data  base. 

The  data  fusion  function  fulfills  the  need  to  provide  data  base 
linkages  for  relevance  of  data  units  comprising  the  same  entity  or 
action.  The  fusion  function  may  be  viewed  as  a  reference  table 
function  allowing  the  collection  of  information  on  one  entity  from 
several  sources  such  as,  voice,  video,  imagery,  and  data  links. 

The  data  fusion  function  shall  fuse  the  conclusions  from  existing  Service 
AAR  tools  to  create  a  Joint  task  completion  as  related  to  the  Joint  task 

The  data  fusion  function  shall  provide  an  Event  Timeline  (any  number  of 
events  can  be  added). 

The  data  fusion  function  shall  provide  a  capability  to  consolidate  virtual 
and  constructive  with  live  range  instrumentation  data. 

The  data/knowledge  distribution  function  fulfills  the  needs  of  the 
training  audience  by  providing  the  right  information  to  the  right 
people  at  the  right  time.  The  AAR  knowledge  network  combines 
repositories  of  information  with  methods  to  organize,  search, 
retrieve,  share,  and  update  the  information  in  the  repositories.  It 
provides  a  common  interface  to  knowledge  that  may  be  contained 
in  multiple  systems  and  stored  in  a  variety  of  formats.  The  KN  must 
include  sophisticated  methods  for  retrieving  useful  information  in  a 
usable  format.  It  must  support  collaboration  among  people  using 
different  and  perhaps  incompatible  information  platforms.  Finally,  it 
must  support  dissemination  of  information  as  well  as  retrieval.  The 
AAR  KN  will  provide  user-friendly  access  to  vast  repositories  of 
existing  information  (and  data)  for  the  training  audience  and  the 
forces  being  trained.  They  will  make  it  possible  to  query  and  search 
these  sources  in  a  timely  and  efficient  manner  and  to  rank  and 
present  the  most  relevant  results  to  the  user.  Figure  three 
illustrates  the  conceptual  need  that  this  function  will  fulfill. 

The  data  distribution  function  shall  be  available  wherever  a  JTEN 
network  node  is  available. 

The  data  distribution  function  shall  be  able  to  archive  local  AAR 
incoming  data  files. 

The  data  distribution  function  shall  be  able  to  archive  all  processed  or 
fused,  analyzed,  or  MOP/MOE  data. 

X  The  data  distribution  function  shall  be  able  to  archive  and  distribute 
knowledge  both  in  real  and  non-real  time.  The  real  time  data  will  be  of 
specific  categories  for  exercise  feedback  for  the  control  of  the  event  and 
for  real  time  casualty  assessment. 
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Functional  Area 


Item  SPOTLITE  Voyager  Analyst 
Edge  Toolkit 


Description 


6.1.7  Analysis 
Function 


6.1.8  Measures  of 
Performance 
(MOP)/Measures  of 
Effectiveness 
(MOE)  Function 

Supports 


Supports 


Supports 


Supports 


6.1.9  Replay 
Function 


6.1.10  Reports 
Function 


6.1.11  Real  Time 
Feedback  Function 


5 

8 

13  X 

14  X 


2  X 

6 


8 

9 


10  X 

11  X 


12  X 

13  X 


7  X 


6 


X 


X 


The  data  distribution  function  shall  continuously  construct  a  distributed 
joint  data/knowledge  database  derived  from  the  fused  information. 

The  data  distribution  function  shall  provide  for  a  distributed  knowledge 
network  that  will  allow  for  searches  stimulated  by  the  operator  or 
automatically  by  an  analysis  or  collaboration  task. 

X  The  data  distribution  function  shall  provide  for  personal  messages. 

The  data  distribution  function  shall  provide  for  system  file  import/export 
to  allow  documents  to  be  brought  into  the  network  easily. 

The  analysis  function  will  provide  the  functional  capabilities  to 
fulfill  the  conduct  of  detailed  analysis  of  sequence  of  actions, 
sequence  of  events,  performance  of  systems  or  subsystems, 
strategic,  operational  or  tactical  communications  performance  and 
COP  commonality  and  distribution. 

X  The  detailed  analysis  function  shall  have  the  capability  to  determine  root 
cause  analysis  of  training  event  anomalies. 

X  The  detailed  analysis  function  shall  have  the  capability  to  determine 
training  event  anomaly  root  cause  identification.  The  planned  scenario 
versus  the  actual  event  conduct  shall  provide  the  information  to  derive 
anomaly  identification. 

X  The  detailed  analysis  function  shall  perform  a  detailed  after-action 
analysis  of  the  mission's  kill  chain. 

X  The  detailed  analysis  function  shall  be  used  to  resolve  situational 
awareness  discrepancies. 

The  MOP/MOE  function  shall  provide  the  capability  to  provide 
subjective  and  objective  information  leading  to  the  assessment  of 
team  performance  during  the  conduct  of  a  joint  training  event.  The 
MOP  and  MOE  data  definitions  and  coefficients  shall  be  preset  into 
the  AAR  tool  set  during  the  planning  function  phase. 

The  MOE/MOP  function  shall  have  the  capability  to  derive  training 
effectiveness  measures,  based  upon  the  execution  of  the  training  event, 
planned  versus  executed. 

X  The  MOE/MOP  function  shall  have  the  capability  to  derive  trend  analysis 
of  training  effectiveness.  The  trend  analysis  shall  identify  from  historical 
training  event  measures,  gaps  in  the  conduct  of  the  training  event. 

X  The  MOE/MOP  function  shall  have  the  capability  to  indicate  gaps  in  the 
performance  and  measure  of  the  training  event. 

X  The  MOE/MOP  function  shall  have  the  capability  to  indicate  gaps  or 
issues  that  span  the  total  doctrine,  organization,  training,  materiel, 
leadership  and  education,  personnel  and  facilities  (DOTMLPF)  domain. 

The  replay  function  shall  fulfill  the  capability  gap  needs  of 
conducting  joint  training  audience  review  of  the  events  actions  at 
the  tactical  operational  or  strategic  level.  Replay  may  be  replay  of 
tactical  screen  captures,  or  video  from  specific  sites,  or  video  of  a 
command  center.  This  function  allows  for  the  training  audience  to 
review  specific  tasks  for  assessment  of  actions  taken. 

X  The  replay  function  shall  have  the  ability  to  replay  Personal  Digital 
Assistants  (PDAs)  non-electronic  data  and  observer  comments. 

The  reports  function  shall  have  the  capability  to  provide  summary 
reports  of  the  events  and  particular  entities  by  time,  space  or 
conditions  dictated  by  operator  inputs.  Also  included  are  the 
capabilities  to  provide  situational  awareness  plots  and  recordings 
for  future  playback  at  remote  sites. 

The  reports  function  shall  provide  tailored  products  within  10  minutes  of 
exercise  completion. 

The  real  time  training  feedback  function  shall  provide  an 
information  feedback  capability  in  near  real  time  of  the  situational 
awareness  of  the  training  grid  for  status  monitoring,  for  real  time 
casualty  assessment  as  denoted  by  the  controlling  operators  and 
general  system  status  of  the  training  event  to  the  operators  and 
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Functional  Area 


Item  SPOTLITE  Voyager  Analyst 
Edge  Toolkit 


Description 


training  audience. 


3  X 

4  X 

5  X 

8 


6.1.12  Collaboration 
Function 


4 

9 

12 

17 

19  X 


6.1.13 

Communications 

Function 


X 

X 


X  The  real  time  training  feedback  function  shall  be  used  for  training 

feedback  control;  thereby  providing  a  near  real  time  playback  capability. 

X  The  real  time  training  feedback  function  shall  provide  the  training  control 
operators  with  the  capability  to  allow  forward  adjustment 
recommendations  based  on  feedback  provided. 

The  real  time  training  feedback  function  shall  have  the  capability  to 
review  live  fire  operations  as  consummated  at  a  MOUT  facility. 

X  The  real  time  training  feedback  function  shall  provide  users  the 

capability  to  establish  filter  parameters  for  delivering  real-time  event 
information. 

The  collaboration  function  shall  have  the  capability  to  provide 
multi-site  collaboration  of  the  event  results  at  simultaneous  sites  to 
critique  results  obtained. 

X  The  collaboration  function  shall  have  the  capability  to  build  one  file  from 
separate  sources  simultaneously. 

X  The  collaboration  function  shall  allow  for  modification  to  stored  observer 
notes  as  desired. 

X  The  collaboration  function  shall  provide  the  operators  the  ability  to 
access  lessons  learned  from  previous  training  events  for  analysis  and 
comment. 

X  The  collaboration  function  shall  provide  for  the  composition  of  conclusive 
JTA  accomplishments  using  both  subjective  information  and  objective 
information  entered  during  and  post  event  activities. 

X  The  collaboration  function  shall  provide  the  users  the  ability  to  construct 
DOTMLPF  recommendations. 

The  communications  function  shall  provide  the  joint  AAR  tool  set 
with  the  ability  to  receive  tactical,  operational  and  strategic 
information  from  various  voice  and  data  links.  The  purpose  of  this 
function  is  to  process  data  link  data  information  in  the  sequence 
and  format  of  the  received  data. 
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Appendix  C:  JMOC  -  JAAR  Measurement  Set 


Prompt 

Type 

Is  time  synchronized  across  all  machines? 

Yes  /  No 

Is  the  DCM  communicating  with  the  data  collectors? 

Yes  /  No 

Is  the  DCM  able  to  create  a  new  event? 

Yes  /  No 

Is  the  DCM  able  to  create  a  new  session? 

Yes  /  No 

Is  simulation  data  being  collected? 

Yes  /  No 

If  applicable,  which  environments  are  being  collected? 

Checklist 

Which  Federated  Object  Model  (FOM)  is  being  used? 

Checklist 

Is  virtual  data  being  collected? 

Yes  /  No 

If  applicable,  which  environments  are  being  collected? 

Checklist 

Is  live  data  being  collected? 

Yes  /  No 

If  applicable,  which  environments  are  being  collected? 

Checklist 

Is  chat  data  being  collected? 

Yes  /  No 

If  applicable,  which  environments  are  being  collected? 

Checklist 

Is  observer  data  being  collected? 

Yes  /  No 

If  applicable,  which  environments  are  being  collected? 

Checklist 

Is  data  processor  cooking  data  for  reports? 

Yes  /  No 

Is  data  analyzer  displaying  units  and  actions? 

Yes  /  No 

Is  data  analyzer  displaying  attributes  of  units  and  actions? 

Yes  /  No 

Is  playback  from  the  data  analyzer  working? 

Yes  /  No 

Is  report  generation  working? 

Yes  /  No 

Does  the  event  agent  report  high  interest  events? 

Yes  /  No 

How  effective  is  the  2D  entity  display? 

Likert  Scale 

How  accurate  is  the  symbology? 

Likert  Scale 

Which  data  sources  (LVC)  are  correctly  identified  by  the  display? 

Checklist 

How  effective  is  the  3D  entity  display? 

Likert  Scale 

How  accurate  is  the  symbology? 

Likert  Scale 

Which  data  sources  (LVC)  are  correctly  identified  by  the  display? 

Checklist 

Is  playback  working  for  2D? 

Yes  /  No 

Is  playback  working  for  3D? 

Yes  /  No 

Is  playback  synchronization  working? 

Yes  /  No 

Is  movie  captured  from  2D? 

Yes  /  No 

Is  movie  captured  from  3D? 

Yes  /  No 

Is  presentation  author  launched? 

Yes  /  No 

Are  the  generated  products  available  in  the  presentation  author? 

Yes  /  No 

Is  the  presentation  author  auto-insert  working? 

Yes  /  No 

Which  products  does  the  presentation  author  successfully  insert  into  the  presentation? 

Checklist 

Are  you  able  to  burn  a  CD/DVD? 

Yes  /  No 

Is  the  operator  capable  of  performing  a  system  startup  and  shutdown? 
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Is  the  operator  capable  of  starting  and  ending  an  event? 

Yes  /  No 

Is  the  operator  capable  of  starting  and  ending  a  session? 

Yes  /  No 

Were  the  operators  able  to  use  the  bookmarks? 

Yes  /  No 

What  map  functionality  does  the  operator  use? 

Checklist 

Does  the  operator  understand  the  use  of  zones? 

Likert  Scale 

Does  the  operator  understand  the  use  of  range  rings? 

Likert  Scale 

Does  the  operator  understand  how  to  navigate  between  multiple  sessions  within  an 
event? 

Yes  /  No 

What  tools  does  the  operator  use  to  navigate  time? 

Checklist 

Does  the  operator  understand  how  to  navigate  to  a  specific  start/end  time  period 
within  a  session? 

Yes  /  No 

Does  the  operator  know  how  to  use  the  report  filter  tool? 

Yes  /  No 

Is  the  operator  aware  of  the  available  reports  and  do  they  know  what  they  mean? 

Yes  /  No 

Does  the  operator  know  how  to  run  the  event  agent? 

Yes  /  No 

Is  the  user  able  to  generate  reports  using  the  High  Interest  Event  (HIE)? 

Yes  /  No 

Does  the  user  understand  how  to  do  synchronized  replay? 

Yes  /  No 

Did  the  JAAR  produce  accurate  reports  for  the  event? 

Yes  /  No 

Was  the  JAAR  able  to  produce  movie(s)? 

Yes  /  No 

Was  the  JAAR  able  to  produce  PowerPoint  presentation(s)? 

Yes  /  No 

Was  the  JAAR  able  to  produce  screenshots? 

Yes  /  No 

Does  the  operator  know  how  to  capture  a  snapshot  from  2D  replay? 

Yes  /  No 

Does  the  operator  know  how  to  make  a  movie  from  2D  replay? 

Yes  /  No 

Does  the  operator  know  how  to  make  and  retrieve  a  quick  report? 

Yes  /  No 
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Appendix  D:  Available  SPOTLITE  Dynamic  Targeting  Cell  (DTC)  Measures 


This  Appendix  presents  some  of  the  measures  that  have  been  implemented  in  SPOTLITE  and  collected  by  various  user 
groups  to  support  training  assessment.  They  are  presented  in  XML  code,  which  is  readily  portable,  and  runs  in 
SPOTLITE. 

These  were  all  developed  through  an  extensive,  structured  process  (called  the  COMPASS  Process,  for  COmpetency-based 
Measures  for  Performance  Assessment  Systems),  working  with  domain  experts.  Each  question  is  tied  to  a  specific 
mission  essential  competency,  for  which  observable  behaviors  are  defined.  When  questions  are  on  a  Likert  Scale,  as 
many  of  them  are,  those  scales  are  anchored  by  specific  examples. 

Similar  measures  are  available  for  a  number  of  different  military  operational  domains,  and  are  in  use  across  Services. 

This  is  header  information  saying  this  is  the  Dynamic  Targeting  Cell  set  of  questions. 

<Module  webEnabled  =  "fa Ise"  appName  = " DTC"  displavGlobalPre- "true" 

displavGlobalPost= "fa  Ise"  title= "  DTC"  caption  =  "S  pot  I  ite  (  DTC  ) "  subcaption  = " " 
buttonPanel  =  "true"  buttonName=  "Ta rget" 

xm  I  ns  :xsi  =  "http://  www.w3.org/2001/XMLSchema-instance" 

xsi :  schema  Location  =  "http://schemas.aptima.com/spotlite  Aspotlite.xsd" 

xm  I  ns= "http://schemas.aptima.com/spotlite"> 

This  gives  subheads  and  points  to  other  palces  in  the  code  and  supporting  materials,  so  the 
questions  can  be  traced  back  to  the  original  Mission  Essential  Competencies  (MECS)> 

-  <Section  name  =  "MECS"> 

-  <Section  treeName  =  "Tarqet"  default="true"> 

-  <Section  name  =  "Identifv  Tarqet"> 

-  <Section  name  =  "PID  Status"  category  =  "questionGroup"> 

-  <Section> 

-  <LikertScaleQuestionUnit  name  =  "51"  showNA="true"  showNO="true"  kevwords="PID 

Status"  MEC  =  "19,17"> 

This  is  the  actual  text  of  the  question  that  appears  on  the  screen.  Note  the  XML  convention 

which  labels  it  "Question  text,"  making  it  relatively  easy  to  find,  even  in  richly  indented  and 
complex  code. 

<Question  text =" Does  the  Target  Duty  Officer  monitor  PID  status  and  updates?"  /> 

This  gives  the  labels  for  the  three  answers: 

-  <Labels> 

< Label  value ="1"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Does  not  monitor  PID  status"  /> 

< Descriptor  text = "Adequately  monitors  PID  status"  /> 

< Descriptor  text= "Always  monitors  PID  status"  /> 
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In  the  balance  of  this  Appendix,  the  XML  is  heavily  edited  to  show  just  the  question  and, 

sometimes,  the  answer  choices.  For  many  of  the  questions,  only  a  "yes"  or  "no"  answer  is 
required,  making  observation  and  recording  relatively  straightforward,  even  in  the  fas-paced 
training  environment. 

<Question  text -"Does  the  Target  Duty  Officer  coordinate  with  the  targets  technician  for 

coordinate  mensuration  preparation?"  /> 

<Question  text =" Does  the  DTC  confirm  that  the  coordinates  are  in  the  correct  format 
(DDD.MM.SS  or  DDDDMM.MMM)?"  /> 

<Question  text =" Does  the  Target  Duty  Officer  request  refinements  to  target  coordinate 
accuracy  when  necessary?"  /> 

< Question  text' =" Does  the  DTC  validate  information  prior  to  making  a  target  promote  or 
nominate  recommendation?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Does  not  validate  information"  /> 

< Descriptor  text = "Validates  only  critical  aspects  of  information"  /> 

< Descriptor  text="Fullv  validates  information"  /> 

<Question  text- "Does  the  DTC  check  or  verify  target  is  not  on  or  in  proximity  to  a  target 

on  the  NSTL  or  RTL?"  /> 

<Question  text="Does  the  DTC  review  target  data  using  the  appropriate  targeting 
manager?"  /> 

<Question  text="Does  the  DTC  query  if  the  target's  set  belongs  to  another  component's 
AO  or  another  target's  set?"  /> 

<Question  text =" Does  the  DTC  acknowledge  or  review  the  emerging  target  tasking?"  /> 

<Question  text =" Does  the  DTC  review  the  notes  within  the  targeting  managers 
(JADOCS)  for  appropriate  mission  details?"  /> 

<Question  text-"Does  the  Target  Duty  Officer  request  refined  or  missing  data  as 
necessary?" /> 

<Question  text- "Does  the  Target  Duty  Officer  request  corroborating  evidence 
(intelligence  fusion)?"  /> 

< Question  text =" Does  the  Target  Duty  Officer  request  additional  collection  when 
appropriate?"  /> 

< Question  text = "If  yes,  from  the  appropriate  person?"  /> 

< Question  text =" Does  the  DTC  place  time  limits  on  requested  data  as  appropriate?"  /> 

<Question  text = "If  yes,  are  the  time  limits  appropriate?"  /> 
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<Question  text="Does  the  DTC  update  the  targeting  manager  when  required?"/> 


<Question  text="Does  the  DTC  prioritize  targets  correctly  (targeting  manager)?"  /> 

<Question  text =" Does  the  DTC  Chief  push  tasking  to  the  team  based  on  target 
perishability?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value =" 2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Pushes  inappropriate  tasking"  /> 

< Descriptor  text="Pushes  mixed  tasking"  /> 

< Descriptor  text="Pushes  relevant  executable  tasking"  /> 

</Descriptors> 

<Question  text =" Does  the  DTC  Chief  effectively  communicate  the  target  perishability  to 
the  team  (i.e.,  target  timeline  estimate)?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text ="Insufficient  communication"  /> 

< Descriptor  text = "Communicates,  but  with  difficulty"  /> 

< Descriptor  text =" Effective  communication"  /> 

</Descriptors> 

<Question  text- "Does  the  DTC  monitor  the  target  nominating  managers  for  developing 
emerging  targets?"  /> 

<Question  text- "Does  the  DTC  Chief  communicate  the  status  of  emerging  targets  to 
affected  players  (i.e.  attack  cell,  SODO,  CCO,  Joint  Components,  etc.)?"  /> 

<Question  text -"Does  the  DTC  use  the  correct  precision  for  the  specified  weapon?"  /> 

<Question  text -"Does  the  DTC  verify  that  the  coordinates  are  correctly  entered  into  the 
targeting  manager?"  /> 

<Question  text="Does  the  DTC  verify  updated  coordinates  from  the  SIDO  are  entered 
into  the  targeting  manager  as  soon  as  available?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value =" 2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Does  not  expedite  update"  /> 

< Descriptor  text="Updates  without  regard  to  prioritization"  /> 

< Descriptor  text = "Correctly  updates  as  prioritized"  /> 
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</Descriptors> 


<Question  text -"Does  the  Attack  Coordinator  begin  to  assemble  valid  weapon  target 
pairings  for  potential  TST  employment?"  /> 

<Question  text -"Does  the  DTC  advise  the  appropriate  personnel  of  changes  in  attack 
priorities?"  /> 

<Question  text="Does  the  Target  Duty  Officer  correctly  employ  JMEMS?"  /> 

This  is  a  follow-up  question: 

<Question  text = "If  no,  what  were  the  point(s)  of  failure?"  /> 

<Question  text- "Does  the  Attack  Coordinator  coordinate  with  the  TDO  in  determining 
appropriate  munitions?"  /> 

<Question  text- "Does  the  Attack  Coordinator  verify  with  TDO  regarding  the  correct 
weapon-target  pairing  for  effects,  environment,  and  threat?"  /> 

<Question  text- "Does  the  Target  Duty  Officer  coordinate  with  the  Attack  Coordinator 
regarding  the  appropriate  terminal  attack  axis,  if  required?"  /> 

This  question  has  a  fairly  complex  follow-up  logic  for  an  answer  of  "no,"  making  it  easy  for  the 
observer  to  refine  the  nature  of  the  observed  error. 

</YesNoQuestionUnit> 

-  <Section  name  =  ""  cateqorv="followUp"> 

-  <Section  cateqorv  =  "followUp"> 

-  <MultipleOptionOuestionUnit  name-"329"  columns^"!"  isMutuallvExclusive-"false"> 
<Question  text = "If  no,  what  was  the  point(s)  of  failure?"  /> 

-  <Options> 

< Option  text="JMEMS"  /> 

< Option  text="CDE"  /> 

<Qption  text-"Deconfliction"  /> 

< Option  text =" Airspace"  /> 

<Ootion  text = "Threats"  /> 

</Options> 


<Question  text="Does  the  DTC  prioritize  attack  options?"  /> 

<Question  text="Does  the  DTC  develop  a  timeline  estimate  for  each  attack  option?"  /> 

< Question  text =" Does  the  DTC  revise  the  attack  planning  priorities  based  on  timeline 
constraints?"  /> 

<Question  text="Does  the  Target  Duty  Officer  make  a  collection  request  that  includes  all 
the  pertinent  information  (EEI,  timeline,  resolution  of  image  or  product,  target 
surveillance)?"  /> 

<Question  text="Does  the  Target  Duty  Officer  coordinate  for  continued  surveillance,  if 
required?"  /> 
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<Question  text =" Does  the  DTC  Chief  reassess  target  dynamics  adequately  enough  to 
maintain  situational  awareness  (SA)  on  the  ability  to  execute  the  attack?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Does  not  reassess  target  dynamics"  /> 

< Descriptor  text= "Reassess  only  critical  target  dynamics"  /> 

< Descriptor  text= "Reassess  all  target  dynamics"  /> 

<Question  text -"Does  the  DTC  coordinate  to  obtain  relevant  target  characteristics,  if 
they  are  not  available  or  assumable?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text= "Irrelevant  data  to  produce  desired  effects  requested"  /> 

< Descriptor  text ="  Relevant  data  to  establish  probability  of  desired  effects  requested" 

/> 

< Descriptor  text= "Complete  data  to  produce  desired  effects  requested"  /> 

</Descriptors> 

<Question  text-"Did  the  Target  Duty  Officer  coordinate  in  a  timely  manner,  consistent 
with  target  priority  and  tactical  dynamics?"  /> 

<Question  text-"Does  the  DTC  consider  non-kinetic  solutions?"  /> 

<Question  text-"Does  the  DTC  check  or  verify  target  is  not  on  or  in  proximity  to  a  target 
on  the  NSTL  or  RTL?"  /> 

<Question  text -"Does  the  DTC  check  targeting  manager  (or  other  approved  source,  e.g., 
brief,  web  page,  etc.)  for  LOAC  restrictions?"  /> 

<Question  text-"Does  the  DTC  communicate  with  SJA  for  LOAC  restrictions  if  not  in 
targeting  manager  (or  other  approved  source,  e.g.,  brief,  web  page,  etc.)?"  /> 

<Question  text-"Does  the  Target  Duty  Officer  collaborate  with  the  Attack  Coordinator 
on  proper  tier-level  processing  for  CDE?"  /> 

<Question  text-"Does  the  Target  Duty  Officer  enter  the  CDE  data  in  the  targeting 
manager?"  /> 

<Question  text-"Does  the  Target  Duty  Officer  accomplish/coordinate  COE,  as  required?" 

/> 

<Question  text -"Does  the  Attack  Coordinator  consider  all  current  tactical  dynamics 
(e.g.,  target  type,  weather,  aircraft  location)  when  developing  attack  solutions?"  /> 
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<Question  text-"Does  the  Attack  Coordinator  consider  all  new  tactical  dynamics  (e.g., 
target  type,  weather,  aircraft  location)  when  planning  ISR  requests  for  attack 
support?"  /> 

<Question  text-"Does  the  Attack  Coordinator  select  an  appropriate  attack  solution 
considering  current  tactical  dynamics  (e.g.,  target  type,  weather,  aircraft  location)?" 

/> 

<Question  text-"Does  the  DTC  coordinate  or  review  an  aircraft/weapons  combination 
that  is  appropriate  for  the  target  given  current  environment  characteristics?"  /> 

<Question  text-"Does  the  DTC  Chief  communicate  target  priorities  based  on  theater 
constraints/tactical  situation?"  /> 

<Question  text -"Does  the  DTC  consider  attack  solutions  based  on  the  target  defense 
characteristics?"  /> 

<Question  text -"Does  the  DTC  review  the  risk  guidance  before  putting  together  the 
strike  package?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Does  not  do  risk  analysis"  /> 

< Descriptor  text ="  Performs  an  incomplete  risk  analysis"  /> 

< Descriptor  text = "Verifies  match  with  higher  headquarter  guidance"  /> 

</Descriptors> 

</LikertScaleQuestionUnit> 

<Question  text-"Does  the  DTC  create  acceptable  package  options  based  on  the  risk 
guidance  for  the  TST  and  its  environment?"  /> 

<Question  text-"Does  the  DTC  review  risk  mitigation  options  (e.g.,  non-kinetic,  stand¬ 
off  munitions,  timing)?"  /> 

<Question  text -"Does  the  DTC  appropriately  coordinate  to  ensure  attack  and/or  support 
assets  to  match  attack  options  with  acceptable  risk?"  /> 

<Question  text -"Does  the  DTC  coordinate  for  target  prosecution?"  /> 

<Question  text-"Does  the  DTC  have  systems  (e.g.,  TBMCS,  JADOCS,  chat,  etc.)  set  up  to 
verify  status  of  all  available  air  assets?"  /> 

<Question  text-"Does  the  DTC  review  scheduled  air  operations  using  the  most  current 
ATO  execution  status?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 


29 


-  <Descriptors> 

< Descriptor  text="Did  not  review  ATO  execution"  /> 

< Descriptor  text="Old  ATO  execution  data  reviewed"  /> 

< Descriptor  text="Most  current  ATO  execution  data  reviewed"  /> 

</Descriptors> 

<Ouestion  text="Does  the  DTC  use  the  appropriate  tools  to  review  the  current  ATO 
execution?"  /> 

<Question  text- "Does  the  DTC  coordinate  with  the  ODO/LNOs  for  scheduled  XINT  and 
SEAD  missions?"  /> 

<Question  text- "Does  the  DTC  use  a  complete  and  current  inventory  of  available  attack 
and  support  assets  for  planning?"  /> 

<Question  text- "Does  the  DTC  review  non-ATO  joint  attack/support  assets  when 
appropriate?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="No  non-ATO  assets  reviewed"  /> 

< Descriptor  text="Some  non-ATO  assets  reviewed"  /> 

< Descriptor  text="AII  possible  non-ATO  assets  reviewed"  /> 

</Descriptors> 

<Question  text- "Does  the  DTC  include  non-ATO  joint  attack/support  assets  in  the  attack 
options  when  appropriate?"  /> 

</YesNoQuestionUnit> 

<Question  text -"Does  the  DTC  coordinate  for  non-ATO  joint  attack/support  assets?"  /> 

Note  alternatives  for  the  following  question  if  a  "no"  is  the  answer. 

<Question  text-"Does  the  Attack  Coordinator  recommend  appropriate  support  assets 
within  an  attack  solution  based  on  tactical  dynamics  (e.g.,  target  type,  weather, 
aircraft  location)?"  /> 

</YesNoQuestionUnit> 

-  <Section  name  =  ""  cateqorv  =  "followUp"> 

-  <Section  cateqorv  =  "followUp"> 

-  <MultipleOptionOuestionUnit  name  =  "323"  columns^"!"  isMutuallvExclusive-"false"> 
<Question  text="If  no  appropriate  recommendation  was  made,  which  tactical  dynamics 

were  missed?"  /> 

-  <Options> 

<Option  text- "Target  type"  /> 
cQption  text="Threats  in  target  area"  /> 

<Qption  text-"Packaqe  time  to  target"  /> 

cQption  text- "Weather"  /> 
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cOption  text="Strike/support  asset  availability"  /> 
cQption  text-"Weapon  effectiveness"  /> 

<Qption  text="Other"  /> 

</Options> 

</MultipleOptipnQuestionUnit> 

</Section> 

</Section> 

<Ouestion  text="If  no  appropriate  recommendation  was  made,  which  support  assets 
were  missed?"  /> 

-  <Options> 

<Qption  text -"Tanker"  /> 

<Qption  text="SEAD"  /> 

<Qption  text="ECM"  /> 

<Qption  text="Other"  /> 

</Options> 

</MultipleOptionOuestionUnit> 

</Section> 

</Section> 

</Section> 

<Ouestion  text =" Does  the  DTC  coordinate  with  the  C2DO  to  obtain  airborne  factors 
affecting  the  plan?"  /> 

<Ouestion  text =" Does  the  DTC  verify  SCL  or  current  weapon  inventory  through 
ODO/unit  LNO  (or  C2DO  if  airborne  assets)?"  /> 

<Ouestion  text =" Does  the  DTC  verify  availability  of  XINT/XSEAD  assets  on  station 
with  tactical  C2  units?"  /> 

<Ouestion  text="Does  the  DTC  coordinate  with  component  command  element  as 
appropriate  for  joint  fires  or  support?"  /> 

<Question  text-"Does  the  DTC  internally  communicate  relevant  information  regarding 
coordination  with  component  liaisons  for  joint  fires  and  support?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="No  relevant  information  communicated"  /> 

< Descriptor  text="Only  the  most  critical  information  communicated"  /> 

< Descriptor  text="AII  relevant  information  communicated"  /> 

</Descriptors> 

<Question  text -"Does  the  DTC  consider  all  critical  factors  when 

assessing/communicating  ATO  impacts  based  on  prosecuting  potential  targets?"  /> 

<Question  text-"Does  the  DTC  Chief's  resource  management  decisions  adversely  affect 
the  efficiency/effectiveness  of  the  overall  planned  ATO?"  /> 
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<Question  text- "Does  the  DTC  communicate  the  impact  of  TST  tasking  on  the  ATO  if 
required?"  /> 

<Question  text- "Does  the  DTC  coordinate  to  ensure  that  BDA  occurs?"  /> 

<Question  text -"Does  the  DTC  Chief  ask  questions  of  his  team  concerning  the  details  of 
the  attack  plan  prior  to  going  to  the  CCO?"  /> 

<Question  text -"Does  the  DTC  Chief  communicate  the  attack  plan  to  the  CCO?"  /> 

<Question  text-"Does  the  DTC  address  the  concerns  provided  by  CCO/SODO  and  replan 
as  required?"  /> 

<Question  text- "Does  the  DTC  Chief  communicate  the  revised  attack  plan  to  the  CCO  if 
required?"  /> 

<Question  text-"Does  the  DTC  verify  attack  complies  with  PID,  CDE,  and  deconfliction?" 

/> 

</YesNoQuestionUnit> 

-  <Section  name  =  ""  cateqorv="followUp"> 

-  <Section  cateqorv  =  "followUp"> 

-  <MultipleOptionOuestionllnit  name  =  "327"  columns="l"  isMutuallvExclusive  =  "false"> 
<Question  text = "If  no,  what  was  missed?"  /> 

-  <Options> 

cQption  text="PID"  /> 

<Qption  text="CDE"  /> 

<Qption  text-"Deconfliction"  /> 

</Options> 

</MultipleOptionOuestionUnit> 

<Question  text- "Does  the  DTC  communicate  recommendations  for  attack  solutions 
using  the  targeting  manager?"  /> 

< Question  text = "Does  the  DTC  select  and  communicate  an  appropriate  attack  solution 
or  recommend  "No-Go"?"  /> 

<Question  text- "Does  the  DTC  Chief  ask  for  final  approval  for  TST/CPT  tasking  from 
required  higher  authority  (CCO,  SODO)  if  required?"  /> 

<Question  text-"Did  the  DTC  coordinate  to  validate  correct  TST/CPT  mission  tasking 
(i.e.,  investigate,  target,  destroy)?"  /> 

<Question  text -"Does  the  DTC  confirm  the  AO  owner  for  battlespace  deconfliction?"  /> 

<Question  text-"Does  the  DTC  coordinate  with  component  command  elements  for 
deconfliction  as  required?"  /> 

<Question  text-"Does  the  DTC  ensure  appropriate  airspace  control  measures  are 
planned  and  communicated  to  Airspace  Management?"  /> 

<Question  text-"Does  the  DTC  input  TST/CPT  mission  tasking  (i.e.,  investigate,  target, 
destroy)  into  the  targeting  manager?"  /> 
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<Question  text="Does  the  DTC  prepare  the  tasking  message?"  /> 


<Question  text- "Does  the  DTC  Chief  coordinate  to  ensure  that  the  team  is  appropriately 
monitoring  mission  execution?"  /> 

<Question  text- "Does  the  DTC  choose  the  appropriate  tools  for  monitoring  mission 
execution?"  /> 

Here  we  see  both  choice  options  and  follow-ups  to  some  of  those  answers,  getting  into 
actionable  detail. 

<Question  text-"Does  the  DTC  coordinate  internally  and  with  external  personnel  (i.e., 
CCO,  SODO,  SIDO,  Plans)  with  regard  to  mission  execution?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="Does  not  coordinate  with  any  appropriate  personnel"  /> 

< Descriptor  text= "Coordinates  only  with  critical  personnel"  /> 

< Descriptor  text = "Coordinates  with  all  appropriate  personnel"  /> 

</Descriptors> 

</LikertScaleQuestionUnit> 

-  <Section  name  =  ""  cateqorv="followUp"> 

-  <Section  cateqorv  =  "followUp"> 

-  <VipersOuestionUnit  name="228"  MEC="17,22,20"  showCGI  =  "false"> 

<Question  text=""  /> 

-  <Options> 

< Option  text="DTC  Chief"  /> 

< Option  text="AC"  /> 

< Option  text="TDO"  /> 

</Options> 

</VipersQuestionUnit> 

</Section> 

</Section> 

</Section> 

-  <Section> 

<Question  text-"Does  the  DTC  maintain  situation  awareness  (SA)  using  the 
appropriate  displays  and  tools  to  determine  mission  result?"  /> 

<Question  text-"Does  the  DTC  review,  assess,  assimilate,  and  act  on  INFLTREP  when 
available  to  establish  follow-on  actions?"  /> 

<Question  text -"Does  the  DTC  request  mission  results  updates  from  the  C2DO  if  TOT 
has  expired  and  no  mission  results  were  reported?"  /> 

-  <Options> 

cQption  text- "Was  the  timeliness  of  the  request  appropriate  given  the  situation?"  /> 
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cQption  text="Was  the  completeness  of  the  request  appropriate  given  the  situation?" 

/> 

<Question  text-"Does  the  DTC  verify/ review  BDA  in  appropriate  targeting  manager  as 
necessary?" /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="BDA  not  verified/ reviewed"  /> 

< Descriptor  text="Onlv  critical  BDA  aspects  verified/reviewed"  /> 

< Descriptor  text="BDA  verified/ reviewed  completely"  /> 

</Descriptors> 

<Question  text- "Does  the  DTC  ensure  that  the  TST/CPT  mission  BDA  is  entered  into  the 
targeting  manager?"  /> 

<Question  text- "Does  the  Target  Duty  Officer  follow-up  with  additional  intelligence 
sources/products  to  corroborate  BDA  as  appropriate?"  /> 

<Question  text- "Does  the  DTC  attempt  to  expedite  closure  of  assessment  post-strike?" 

/> 

<Question  text- "Does  the  DTC  Chief  check  the  targeting  manager  for  currency  of 
mission  status?"  /> 

<Question  text -"Does  the  DTC  coordinate  on  the  mission  results  to  decide  on  future 
actions  (i.e.,  request  additional  intelligence,  mission  complete,  re-strike,  pass  to 
Plans)?" /> 

<Question  text- "Does  the  DTC  close  out  actions  on  target  appropriately  once 
assessment  is  completed?"  /> 

<Question  text-"Does  the  DTC  correctly  communicate  actual  assets  used  versus  planned 
assets  to  the  appropriate  personnel  (SODO,  ODOs,  Plans,  etc.)?"  /> 

<Question  text-"Does  the  DTC  review  updates  to  relevant  guidance  (e.g.,  documents, 
messages,  verbal,  chat,  tables,  AOD,  MAAP  brief.  Commander's  guidance)?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="No  relevant  guidance  reviewed"  /> 

< Descriptor  text="Onlv  critical  guidance  reviewed"  /> 

< Descriptor  text="AII  relevant  guidance  reviewed"  /> 

</Descriptors> 
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<Question  text="Does  the  DTC  Chief  make  the  DTC  team  aware  of  updates?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  valuer "2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text= "Never"  /> 

< Descriptor  text = "Sometimes"  /> 

< Descriptor  text= "Always"  /> 

</Descriptors> 

</LikertScaleQuestionUnit> 

-  <Section  name  =  ""  cateqorv="followUp"> 

-  <Section  cateqorv  =  "followUp"> 

-  <MultipleYesNoQuestionUnit  name  =  "7"  showNA= "true"  showl\IO  = "true"  columns^ "  1" > 
<Question  text="Updates  to:"  /> 

-  <Options> 

<Qption  text="SPINS"  /> 

<Qption  text =" ROE"  /> 

cOption  text-"No  Strike  and  Restricted  Target  List"  /> 

cQption  text = "Weather"  /> 

<Qption  text="TST  Matrix"  /> 
cOption  text-"Other  updates"  /> 

</Options> 

</MultipleYesNoOuestionUnit> 

</Section> 

</Section> 

</Section> 

-  <Section> 

<Question  text="Does  the  DTC  request  further 
guidance  as  appropriate?"  /> 


<Question  text =" Does  the  DTC  Chief  present  complete  and  accurate  information 
regarding  guidance  changes  to  the  team?"  /> 

<Question  text="Does  the  DTC  Chief  ensure  that  the  team  understands  each  of  their 
roles  and  responsibilities  in  the  dynamic  targeting  process?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text= "Assumes  team  understands  with  no  verification  or  pre-brief"  /> 

< Descriptor  text= "Questions  team  members  only  on  critical  aspects  of  dynamic 
targeting"  /> 

< Descriptor  text =" Proactively  establishes  roles  and  responsibilities  with  pre-brief  and 
verbal  assessment"  /> 
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</Descriptors> 

</LikertScaleOuestionUnit> 

</Section> 


<Question  text-"Does  the  DTC  have  all  applications  and  resources  open  and/or 
available  for  efficient  retrieval  of  critical  information?"  /> 

-  <Labels> 

< Label  value="l"  /> 

< Label  value="2"  /> 

< Label  value="3"  /> 

< Label  value="4"  /> 

</Labels> 

-  <Descriptors> 

< Descriptor  text="None  open  and/or  available"  /> 

< Descriptor  text = "Critical  apps/ resources  open  and/or  available"  /> 

< Descriptor  text="AII  relevant  apps/ resources  open  and/or  available"  /> 

</Descriptors> 

</LikertScaleQuestionUnit> 

-  <Section  name  =  ""  cateqorv="followUp"> 

-  <Section  cateqorv  =  "followUp"> 

-  <MultipleOptionOuestionUnit  name  =  "18"  columns="l"  isMutuallvExclusive  =  "false"> 

< Question  text =" Which  applications  or  resources  were  not  open  and/or  available?"  /> 

-  <Options> 

<Option  text-"Chat  windows"  /> 

<Qption  text="COP"  /> 

<Ogtion  text= "Targeting  manager"  /> 

<Qption  text="CDE  Tool(s)"  /> 

<Option  text-"Tarqet  manager"  /> 

<Qption  text-"Current  ATO  ISR  asset  availability"  /> 
cQption  text-"Current  SITREP  for  Area  of  Responsibility"  /> 
cQption  text-"Current  ATO  Attack  asset  availability"  /> 

<Qption  text-"Weapons/Tarqet  pairing  guidance"  /> 

</Options> 

<Ouestion  text-"Does  the  DTC  review  and/or  refresh  relevant  applications  when 
appropriate?"  /> 

<Question  text- "Does  the  Attack  Coordinator  select  the  appropriate  common  operating 
picture?"  /> 

<Question  text- "Is  the  Attack  Coordinator  aware  of  current  IPB/Intel  situation?"  /> 

<Question  text-"Does  the  DTC  communicate  with  personnel  when  there  are  information 
shortfalls  and/or  needs?"  /> 

<Question  text-"Does  the  DTC  up-channel  chronic  resource  shortfalls  to  the  appropriate 
personnel?"  /> 

<Question  text-"Does  the  Target  Duty  Officer  up-channel  collection  request  shortfalls?" 

/> 
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<Question  text="Does  the  DTC  populate  collaboration  tools:"  /> 


<Question  text- "Does  the  DTC  prioritize  workload  in  a  manner  that  can  acknowledge 
new  emerging  targets?"  /> 

<Question  text- "In  the  event  of  a  communication  failure,  does  the  DTC  use  the 
appropriate  backup  systems?"  /> 
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Appendix  E:  Summary  of  Digital  Pen/Paper  Technologies 

Digital  Pen  and  Paper  technology  has  been  available  commercially  for  about  five  years. 
While  the  technology  continues  to  advance  it  has  not  made  the  significant  breakthrough  that  has 
allowed  it  to  make  large  in-roads  into  everyday  use.  The  problem  lies  with  the  technology  and 
the  current  solutions.  The  basic  approach  behind  digital  pen  technology  is  to  track  the  position  of 
the  pens  tip  and  then  store  the  coordinates  as  a  digital  file.  The  digital  file  is  then  interpreted  and 
processed  through  optical  character  recognition  (OCR)  software  once  the  files  have  been 
transferred  to  a  standard  computer.  Transfer  is  done  via  a  universal  serial  bus  (USB).  In  order  to 
track  the  pens  location  and  establish  the  reference  points  necessary  to  plot  and  record  the  pens 
movements  three  different  techniques  are  currently  being  employed: 

1 .  Digital  Pen  with  special  paper 

2.  Digital  Pen  with  special  transceiver 

3.  Digital  Pen  with  transceiver  tablet 

1 .  Digital  pen  with  special  paper 

The  technique  used  here  is  to  have  either  dots  or  a  grid  printed  on  paper  that  the  pen  can 
reference  to  help  determine  its  location.  In  most  instances  the  dots  are  printed  on  plain  paper 
so  the  cost  of  the  paper  is  minimal  although  the  purchase  of  this  pre -printed  paper  becomes  a 
necessary  adjunct  to  the  use  of  the  pen.  One  manufacturer  provides  a  portable  data  file 
formatted  (PDF)  file  containing  the  necessary  dots  so  that  the  user  has  the  option  of  printing 
their  own  paper.  The  pen  with  pre-printed  paper  appears  to  be  the  most  popular  technique 
and  the  paper  now  has  a  technical  specification  and  is  referred  to  as  Anoto  Paper  or  Anoto 
Technology.  Many  of  the  major  paper  manufactures  -  including  3M,  Franklin  Covey,  and 
Oxford  -  now  produce  Anoto  paper,  (www.anoto.com). 

Pens  using  this  technology  include: 

•  Logitech  io2  (purchased  for  testing) 

•  Nokia  SU- IB 

•  Magicomm  G303 

•  Maxell  Penit  Digital  Pen  (appears  to  be  pen  used  by  Magicomm  above) 

•  Sony  Ericsson  Chatpen  CHA-30 

2.  Digital  Pen  with  special  transceiver 

In  an  attempt  to  free  the  digital  pen  from  requiring  special  paper  or  a  special  tablet  a  few 
companies  have  developed  a  special  transceiver  device  that  can  be  clipped  to  the  paper  or 
clipboard  the  paper  is  being  used  on.  This  transceiver  then  communicates  with  the  pen  and 
tracks  the  pen's  position  and  records  the  data  in  the  transceiver.  The  data  is  then  transferred 
from  the  transceiver  to  a  computer  via  USB.  The  advantage  of  this  approach  is  the  freedom 
to  use  it  anywhere  with  no  special  paper  or  tablet.  The  disadvantage  is  that  there  can  be 
interference  between  the  pen  and  the  transceiver  especially  by  left-handed  writers  whose 
tendency  to  curl  their  hands  above  the  pen  often  blocks  or  interfers  with  the  transmission. 
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From  reviews  it  appears  that  this  approach  is  not  as  accurate  as  the  pen  and  paper  approach. 
Pens  using  this  technology  are: 

•  Pegasus  Mobile  NoteTaker 

•  EPOS  Digital  Pen 

3.  Digital  Pen  with  transceiver  tablet 

In  this  approach  the  pen  transmits  its  location  to  the  tablet  which  records  the  data.  The 
data  is  then  transferred  from  the  tablet  to  a  personal  computer  (PC).  An  advantage  of  this 
approach  is  that  no  special  paper  is  required  and  the  tablet/pen  combination  should 
theoretically  be  the  most  accurate.  The  disadvantage  is  the  need  to  use  and  transport  the 
supplied  tablet.  Reviews  of  the  current  products  using  this  technique  tend  to  indicate  that  the 
construction  of  the  products  is  on  the  flimsy  side. 

Pens  using  this  technology: 

•  ACECAD  DigiMemo 

•  Adesso  Cyberpad  (purchased  for  testing) 


Summary  of  Reviewed  Software 

Almost  all  of  the  hardware  manufactures  above  rely  on  third  party  software  for  the  OCR 
translation  of  their  handwriting  captures.  The  one  exception  is  Logitech,  which  provides  its 
own  software  but  is  also  compatible  with  MyScript  Notes. 

1.  MyScript  Notes:  This  appears  to  be  the  leading  handwriting  recognition  software  and  is 
compatible  with  most  products  listed  above. 

2.  EverNote  ritePen:  This  is  another  handwriting  recognition  software  application  that  is 
compatible  with  many  of  the  products  above.  It  does  not  appear  to  be  as  highly  rated  nor 
as  extensible  as  MyScript 

3.  CalliGrapher  (for  pocket  PCs):  This  is  the  leading  handwriting  recognition  software  for 
Pocket  PCs. 


Summary  of  Digital  Pen  Technologies 
Advantages: 

1 .  This  method  allows  the  transference  of  the  observers"  input  into  digital  format  with  the 
least  effort. 

2.  It  provides  the  most  familiar  solution  as  it  basically  allows  the  observers  to  continue  with 
their  current  method  of  using  hand  writing  as  an  input  source. 

Disadvantages 


39 


1 .  It  does  not  allow  for  any  feedback  on  the  ability  of  the  software  to  decipher  and 
translate  the  input  text.  In  some  instances  the  translations  might  not  work  well  and 
manual  input  of  the  data  may  be  required.  Since  all  pen  data  is  in  essence  backed  up 
by  a  hard  copy  paper  version  this  is  not  necessarily  a  major  issue  although  the  use  of 
some  personnel  for  review/cleanup  as  well  as  auxiliary  entry  may  be  required. 

2.  It  does  not  allow  the  observer  to  receive  any  guidance  or  feedback.  This  is  basically  a 
one  way  procedure  from  the  observers  to  the  computer  system  with  no  opportunity 
for  any  guidance,  feedback  or  auxiliary  information  to  be  provided  by  the  computer 
system. 

3.  Data  must  be  transferred  into  the  system.  While  a  pocket  PC  or  tablet  PC  can  be  set 
up  to  directly  transfer  data  into  the  computer  system  with  the  proper  tags,  data 
captured  and  then  translated  into  the  system  will  still  need  to  be  added  to  the 
database. 
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